To add on to this, in front of other people, I play with my finger ring that has moving parts. It's a lot more descreet than playing with other objects.
Fidget rings are great but it's hard getting them in metal I can wear, since I break out with most cheap steel rings. Had one in silver but since silver is pretty soft the ring isn't circular anymore but the fidget bands are and they slip off.
I stroke my beard constantly. At least stroking your chin like that is associated with thinking so it's not super weird looking. Unless you're doing it like this:
I have trichotillomania (hair pulling) but only on my beard/goatee. Your post made me think people must look at me funny too, but I bet they don't even notice.