I've been asked if I smoke cause I think they'd ask me to come with but I say no, and no one's gonna ask you to come with them just to watch them smoke lol
I don't smoke, tobacco nor cannabis, but I've had friends who've smoked both.
And when they go on a cigarette break, which they inevitably do, I just say I'll tag along, or if they ask if I smoke, or want a smoke, I'll say "Nah, but I'll join you out there" (which is usually understandable, compared to staying behind on your own).
If they're going to a smoking shelter or stood outside, then they're usually perfectly fine with you standing a few steps away, and my friends have always been quick to move and accommodate me if their smoke is being blown towards me (I've never even had to ask).
And while smokers tend to regularly smoke because they have to, it's also a social thing. You're still friends with them, so you can still chat and have a natter, like normal. It won't just be a case of you watching them smoke. They take breathes in between puffs, so they still speak.
If you can put up with the smell, and occasionaly being subject to some second-hand smoke, then you should just join your friends if they go to smoke.