Time travel within the same universe is not possible, it is a fun fiction which is always contradictory in some way. The only time travel possible would be the one that William Gibson uses in The Peripheral. His idea is that every time you go back in time a new parallel universe is created, and it doesn't impact your current universe because of that.
My theory is that we're one of the most advanced species in our galaxy, and yet we still can't reach another solar system. The probability of intelligent life forming from unintelligent life is extremely unlikely, and we had life on Earth for a LONG time before humans evolved. Intelligent life is very difficult to form, you need the perfect conditions and perfect stressors over millions of years. Then on top of that intelligent life which can reach another solar system is even less likely.
There's life out there thinking the same thing right now:
One of my favourite theory about human evolution is the stoned ape theory. It delivers the conditions of the evolution. Apes were forced outside of the jungle and ventured into open field and had to depend on different nutrition. So they ate some mushrooms and eventually ate ones with psylocibin. Small amounts increase your eyesight - so you can hunt better, bigger amounts are very sexuall arrousing - so more reproduction :)
You don't even need that hypothetical for it to make sense. Look up how chimpanzees fight against each other for territory. It's survival of the fittest over millions of years, simple. Cold weather in the north, requiring the development of technology for survival, learning how to use fire to cook clean food, learning how to use alcohol to purify water, etc. We split from chimpanzees millions of years ago, and chimpanzees can only survive in very lush jungle areas because of the food sources they rely on, which is why they're endangered.
Essentially the type of time travel employed in Harry Potter and the Prisoner of Azkaban. So you wouldn't be able to change anything, even if you did do things — as if you did something, it had always already been done previously (before you travelled back).
"it's a fun fiction which is always contradictory in some way" <-- that is not true.
General relativity allows for closed time-like curves and the existence of those means time travel isn't necessarily impossible, and if that is the case, it has to abide by the Novikov self-consistency principle.
Dumbledore disagrees. The type of timetravel that happens in HP: Prisoner of Azkaban is self-consistent. Same as with Rick's timetravel in the snake episode of Rick & Morty (specifically excluding the snake time travel, which is an example of non-consistency, leading to endless paradoxes).
Now ofc having a timeturner and/or a timemachine would be the impossible part there, but again, CTC's are technically allowed by GR.