I love my cat and she's very affectionate overall, but she just will not sleep on me. She has to be in the same room as me while sleeping, but always on a chair or on the floor in the corner. I guess I thrash around too much in my sleep for her tastes.
My sister's cat is exactly the same. But she smothers him with affection. I was laying on her couch, when I said I noticed her cats avoiding laying/walking on people. Lo and behold 5 min later that cat came up to me. I ignored him, and he came up to nap on me. She said that never happens. I say give them space.