What? Don't get off the bus before you're supposed to just because some stranger is wrong about where they think you're getting off. Quit being a door mat.
Yeah, this is less about being a push over and more about staying safe. Personally, I'd start with just ignoring them, but if they get insistent, I'd do what I could to get away from them. Some people are messed up.
My first time thought went to it being some psycho about to shoot up the bus but for some reason having decided you shouldn't be part of the slaughter. "No no, this is your stop."