let's look at the locations for where the Master Sword resides, the first being The Temple of Time in Ocarina of Time. This is where we learn that the Temple of Time was built to house the Master Sword. If this is the temple of time in Skyward Sword then it would be a conflict in story and timeline and I don't think Nintendo would make that mistake so easily... The next location I'll address is Twilight Princess. The Master Sword is located in the ruins of the Temple of Time, which have now been taken over by forest. While he does travel through a portal to go into the old temple of time, the one that's not falling apart, it is still in the same location none-the-less, just at different times. The next resting place of the Master Sword is in A Link to the Past. This sword is also found in the forest with even less building and protection around it. Now I'm not going into timeline theories here I'm just noting the details of where the Master Sword is located throughout the games. Now... the last area, and most confusing to me, is Wind Waker, where the Master Sword is found UNDER Hyrule Castle... In the other games it's been exclusive to the Temple of Time or its remnants, but in Wind Waker it's in a totally different location as if it's been moved or the Castle's been moved. I'm not sure which. The point I'm making with the Master Sword's resting place is that it's always been in the Temple of Time, WW aside, and the temple itself is in best condition in OoT, making the Temple of Time in SS a conundrum at the least.
Also when you compare Twilight Princess's and OoT's world maps, just about everything matches to a degree, with some slight alteration with the forest and castle, noting that the castle is now in Central Hyrule and not in Northern Hyrule. The other noticeable difference is Zora's Domain, but this is irrelevant. Also when you match up the map with Wind Waker's remaining islands, they are close as well, assuming Dragon Roost is Death Mountain. This would make Wind Waker's and Twilight Princess's maps extremely close too with having the castle located near the center of Hyrule. Taking the location of the castle and moving it to the center of the nation seems to be the end result when you look at any game stemming from OoT that includes the Master Sword and if I had to guess, speculation of course, that would mean that forest became overly dominant and took over the area where the castle was originally. But taking into consideration that Death Mountain is never overgrown... it is a shaky idea at best... but alas it is only my observation and guesses.