Which is farther: the distance from here to here, or the distance from here to there? I say it is from here to here. Because in order to get to here from here you have to go there, then come back to here. Whereas if you're already there, all you have to do is come here.
But why, you ask, can't you just stay where you are if you're already here? Because here refers to a spacetime location. If you are in space you are also in time. There is no purely spatial reference. That is basic relativity: we live in a four-dimensional universe. There is no "getting here" without going through time. So however close to here you designate there to be, it will always take you twice as long to get from here to here.