Originally, the Temple of Time was supposed to have an area that lead to the rear of the Lost Woods. That was way back in one of OoT's beta stage and isn't really that relevant here, but it is interesting to note before I give my actual reasoning. And remember, this is all left up to opinion. Nothing has been proven about this, but I'll give the most logical explanation for why things show up the way they do here.
Like Jesper said, things change. The period between OoT and TP was 100+ years and what I see happening is that the Lost Woods has grown over this period. If you take OoT, then you have TP next in line on that part of the timeline, then later on down the road you have ALttP, it makes perfect sense. In OoT, the Temple of Time was standing perfectly normal. 100+ years later in TP the Temple of Time has began to fall apart and only some rubble is now visable. No explanation for why it is falling apart but we know that it is. We also know that by this time the Lost Woods have began to grow around the Temple of Time. Later on, by the time of ALttP, we see that the Temple of Time is no longer visable at all, but the Master Sword is still in the same spot within the lost woods.
Basically, over the vast number of years between OoT and ALttP, we see that the Master Sword and its pedestal never change location, but the Temple of Time eventually disappears and the Lost Woods grow around it. TP just gives us a nice middle ground visual of this happening between those two titles. As for Hyrule Castle, the only thing you can logically say is that it moved over time. In OoT, it was right next to the Temple of Time. In TP, it was way north of the Temple of Time. In ALttP, it is south-east of where the Temple of Time once stood (the Lost Woods/Master Sword location). It doesn't make sense to say that the Temple of Time moved, but it does make sense to say that Hyrule Castle did.