That's like somebody saying in 1912 that fax machines could never be invented because no printouts were magically appearing on their desk. The technology had to be invented before it could be used. If a time traveler has to step out of a machine, that machine has to be invented first. The idea is that backwards time travel would only be able to travel as far back as the invention of backwards time travel.
That being said, from a physics standpoint I can absolutely see backwards time travel as being impossible. We can't move negative distances across spatial dimensions, so why would we be able to move backwards in time?
That's what I think is so neat about time travel. As a rational concept it's so outlandish and ridiculously unlikely. Everything we know about physics says "Nope."
But the concept to the human imagination seems to be such a natural and easy to grasp fantasy.
Perhaps borne from our desire to fix mistakes, prevent tragedies, and "be in the right place at the right time."
On the simplest level we learn best through our mistakes, but wish dearly we could hang on to that knowledge while hitting "Ctrl + Z" on the consequences.