This. I like that Dr who actually has had this problem in universe. I don't recall the episode, but he went to earth and ended up at the right time, but not the right place, since you know, earth is moving.
Even if you were to use the sun as a reference we orbit the sun (relative to the position of the sun) at some incredible speeds. Time of day factors in, since we're rotating rather fast as well. So getting the right coordinates in space for a particular day, and a particular time in a particular year, for a specific place.... Well, good luck.
Which isn't to mention the fact that we're in a galaxy, which is moving as well, so using a point of reference outside the solar system becomes insane to try and calculate; which is what you would have to do in order to enable travel outside of our solar system with something like a TARDIS.