Unsure how to break the touch barrier

Whenever I see someone I'm interested in I always make sure I go and talk to them. That's as far as I've ever gotten.

The way I see this working is as follows:

1. somebody catches my eye
2. I go over and talk to them
3. we get along well, stuff develops in pretty much the same way as if I had just met a new platonic friend
4. ???
5. We start holding hands. I've watched enough films to see that it pretty much escalates by itself from there.

The problem is that whenever I've done this, they were either cool but didn't show much of an interest in me, or their personality didn't resonate too deeply with mine which was a shame because I still thought they were gorgeous.

Now I'm not looking for somebody to spend the rest of my life with. Because that will take a lot of meeting people. But I am in the mood to experiment with intimate relationships, and now. Part of me wonders whether it's even worth it if they don't share my sense of humour. But another part of me thinks the steps above might be constraining me to only one type of relationship, those of the lifelong sort, which is why it's taking so long.

As you can see at step 4 there is clearly a gap between talking with them and holding hands that I don't know how to cross, which I'd currently do by explicitly asking can we hold hands. I wonder if the thing I'm missing is also the thing that would progress things to the physical without the person being your soulmate. When you go to parties you see drunk people breaking the touch barrier together without talking. What's the cue for that to happen? Should touch ever be the thing that advances a relationship with someone? How does that work? How do you make sure it's mutual? Or is the way to go really to wait until I meet someone I get along with so well that something clicks?