I have the same experience. I act very carefully around them not to threaten them. I also put a tiny bit of my food on the side for them when they get interested - I love watching them eat. They're like little insect tigers - striped, fierce, but tiny!
They don't do that with me. A wasp stung me once because it was in my shoe, so I was obviously perceived as a threat when trying to put it on. I think there was another time but I don't remember, I might've touched it first as well. The rest of the time, wasps seem to respect me, and it's mutual. I've had wasps centimetres away from my face, but I never flinch and I've never regretted not flinching. Took more hits from people trying to kill wasps than from the wasps themselves.
And I've had a wasp sting me just because I deserve to get fucked, I suppose. It just flew up, landed on my hand, sting me, then fucked off back to whichever circle of hell whence it emerged. There were dozens of other people around, but the allergic teenager was the only one who needed to have their weekend ruined.
I was camping with some friends ( all around 13 years old ) and one of my friends, the only allergic one in the group, sat on a wasp nest that was attached to a piece of trunk. The poor guy was stung all over. Luckily we were nearby a hospital and we were able laugh it off a few days later.