Eh, it's been touched upon. Most people argue that they are either the Gerudo or the Sheikah.
On a personal level, I feel that they are more likely the Gerudo, as Impaz is a Sheikah and is
not banished. We weren't told exactly when Midna's ancestors were banished, only that it was long ago. However, Ganon was banished long ago (over a century before TP), so it's possible that the banishment of Ganon resulted in the banishment of his people, making them Midna's ancestors. But who knows.
One thing that does help support the Gerudo idea is that the Dark Interlopers were said to have dark powers. Ganondorf had dark powers (even without the Triforce of Power), and Koume and Kotake had said powers as well. In other games it's also mentioned that Ganondorf and his followers had powers. So maybe it's the case.
People who support the Sheikah idea like to point out that there is the Sheikah Symbol in the Twilight Realm and on Midna's helm, but I like to point out that Zant and Midna's clothing carries Gerudo patterns.
One thing I do like to point out is this scene:
http://www.youtube.com/watch?v=VRa14eIa750
Everything in that scene is symbolic, but what does it symbolize? Mostly, how greed affected the people and blah blah blah. But what I like to point out is that it really,
really puts some of ALttP's backstory into picture:
"In the gathering twilight, the Triforce shone from its resting place high above the world. In a long running battle, the leader of the thieves fought his way past his followers in lust for the Golden Power. After vanquishing his own followers, the leader stood triumphant over the Triforce and grasped it with his blood-stained hands."
The imagery shown in the video is mainly meant to represent how greed and lust for the Triforce blinded people and their impurities caused chaos. However, I also think it was a slight nod to an old story. The part where Ilia is willing to stab Link and then instead Link kills her and tries to take the power for himself is the part where it represents all that. Likewise, it could also represent the old story of how Ganon was also one of these people who'd betray his own people out of greed.
But really, there is no solid evidence of either side of the argument, so at this point all we can do is ponder the possibilities.
