The Master Sword in OoT didn't lead to the Sacred Realm the way you think. The Sword itself has no connection to the Sacred Realm, it was just that drawing it from the Pedastal of Time was the last barrier between the worlds. The bridge between the Sacred Realm and Hyrule is in the Temple of...
The Four Swords gives Ganon access to the Sacred Realm... that's a huge stretch. Regardless, it's commonly held that Four Swords Adventures would be a distant sequel to A Link to the Past even by theorists who don't adhere to the Official Timeline. Ganon in the game is a reincarnation of AlttP...