first, and less importantly, your wheels are gyroscopes
second, and much more importantly, at speed you use your steering to compensate for imbalance. You lean a little right? slight steering to the right compensates. When standing still, steering is no longer an option (duh)
Can confirm. Last week, I got home from a ride, stopped in front of the garage, couldn't unclip, and promptly fell over. It turned out one of the bolts fell out from the cleat during the ride, so the cleat just rotated, instead of unclipping. D'oh. Fortunately, I mostly landed in grass, though I did scrape my ankle a bit.
Same principle as a gyroscope: a turning wheel will tend to stay perpendicular or parallel to the direction of the gravity vector because if it starts tilting away from such orientation there's a force that pushes it back.
Also works better with bigger wheels (if I remember it correctly the effect is related to spinning momentum).
I was pretty surprised when learning Physics and they show us how to derive the formula for that (which I totally forgot since that was over 3 decades ago).
Edit: Actually the gyroscopic effetc is just a part of it. See this article
I'm surprised how much I'm seeing gyro brought up in these comments. It's a factor, but it's practically negligible. It's all in the steering. Start to tip right, and you'll subconsciously steer slightly to the right to correct your balance. Try to ride as slow as you can and you'll find yourself doing these corrections much more frantically and dramatically. The reason for that is because it takes longer for the wheel to roll under your center gravity and "catch" you when you're going slowly so you have to turn in quicker to maintain balance.
Notice that on almost every bike you see, the front axle on the bike is slightly ahead of the neck's axis of rotation. That offset does two things: 1. It stabilizes the steering so that the bike will tend to steer straight and 2. (more important to my point) It makes the balance-correcting effect of steering more immediate and dramatic, making it much easier to ride at slower speeds.
As a counter argument showing why gyro is barely a factor, these exist:
Edit: if you're not seeing the image like I'm not, Google "ski bike".
I don’t get it when people (usually chavs) can just sit back with their hands in their pockets - when I try it my handlebars twist out to one side instantly.