I'll clear up a few things here:
I'll admit it HAS been a long time since I played LttP (and I've played a whole lot of games since then), but I DO remember that the mirror only goes from the Dark World back to the Light World. To go the other way, you need to find a portal that's already in existence or go back to the portal created when you leave the Dark World. So, it's not THAT easy for any other powerful being to enter the sacred realm. They could never influence it like Ganon did, either, because Ganon turned it into the dark world by using the Triforce, which is never easily acessible.
About the FSA->LttP thing:
Ganon does get sealed in the Four Sword and does NOT get sealed in the sacred realm - at least, not automatically. LttP (only in the GBA version) shows us that the Four Sword and it's shrine were somehow moved to the sacred realm.
I know there's some four Dark Links or something (I'm not sure; I haven't played it) in the Four Sword Shrine in the GBA LttP, so it's possible that they released Ganon. After all, the Four Sword is (I'm pretty sure) split into four pieces at that point. As for how they got there... we know that Dark Link is in FSA. According to Zelda Wiki, when you finally defeat the Dark Links in that game, "Zelda undoes the seal on the Dark Mirror and takes it into her possession, dispelling them." At the end of the game, we see Zelda go through a doorway, quite possibly into the Four Sword shrine to leave the mirror with the sword. After the shrine is taken into the Sacred Realm, the seal on the mirror (not the same mirror as LttP's mirror) is obviously broken somehow, possibly by a monster. The fact that there are four Dark Links in the Four Sword Shrine can be attributed to the fact that there are so many of him in FSA, and because there are four Links right up until they put the Four Sword back. It makes sense that Dark Link has the motives to release Ganon - he'd had such motives since the very beginning of FSA, where he forces Link to draw the Four Sword and release Vaati.