I have been thinking on this, lately. There seems to be a whole thing with masks with the people from Lorule, Termina, and the Twilight realm. I have already theorized that Termina is withing the realm that Lorule is located in. I am wondering if the interlopers may have came from there, tried to take Hyrule's Triforce, because they no longer had one, then were banished.