I once started a thread inquiring as to when the Interlopers were banished from Hyrule. The general consensus put the event sometime during or after the Hylian Civil War - perhaps even as a culmination of the war as part of the King of Hyrule's efforts to unify the races. I presume that the King rallied the races against the Interlopers as part of unifying the country and eliminating the biggest threat to it in the process. Once the Interlopers had been banished, an alliance was struck between the Hylians and the Goron and Zora tribes and presumably the Gerudo as well, if even a tentative one in the latter's case at least.
Therefore, the Interloper's banishment would have been pretty recent to the events of Ocarina of Time. I always got the sense that a lot of the sick and twisted stuff around Hyrule stemmed from the recent civil war. The undead beneath Hyrule field, the Forest Temple and yes, the Shadow Temple.
Shadow and Twilight are practically synonymous with each other. Coincidence? If the Interlopers were a dominant threat, then their presence in the Shadow Temple could have been significant. It could have been a dungeon holding and processing a large number of Interloper POWs and it could be that the Interloper's negative energy accounts for a large portion of the temple's evil.
However, it is unknown to what extent the Interlopers were affiliated with darkness and shadow before being banished to the Twilight Realm. It is strongly implied if not outright stated that the Twilight Realm is not the Interloper's place of origin, and that the Interlopers became the Twili after exposure to the twilight. If darkness or twilight was their element before their banishment, then it could imply a connection to the shadow temple. But we simply don't know this. The only solid evidence I can think of to suggest that the Interlopers were in any way associated with darkness is the manner in which they are metaphorically depicted in the Light Spirt's vision to Link as doppleganger's of Link with dark shadowy complexions.
