Yes, Ganondorf was banished. But the Dark Interlopers are plural because there is an "s", meaning that there was more than one person banished.
"Among those living in the light, interlopers who excelled at magic appeared. Wielding powerful sorcery, they tried to establish dominion over the Sacred Realm."
So, yes, Ganon did try to establish dominion over the Sacred Realm, but many people did. In A Link to the Past, people who sought the Golden Power (Triforce) were trapped and twisted in the realm. Not all of them were Ganon's followers. An example of this is the Old Man's grandson (flute boy) who disappeared while searching for the Triforce.
As for the "3 people", that was merely an iconic depiction. The use of Dark Link model was symbolic of peoples' darker, greedier sides. The name "Dark Interlopers" is probably not the actual name of those banished, but a descriptive title. An interloper is somebody who puts themselves into others' affairs, and dark is obviously meaning the opposite of good. The three we see in a cutscene should not be taken literally... that'd be as ridiculous as taking the image of Ilia trying to kill Link with a knife. That entire scene is a representation of people blinded by greed and the power of the Triforce.
None of that, however, has ANYTHING to do with ANYBODY from Majora's Mask. Likewise, there is NO mention of the Sheikah. There is absolutely no mention of people from another world. The only thing that can be considered from that is the term "Interloper"- If they were interfering in affairs other than their own, why were said affairs not their business? But we can be almost sure that these "interlopers" were of Hyrule, as they are said to be "among those living in the light". The fact that they are referred to as "dark" beings who live in the "light" emphasizes that they are a wicked people.
It's said there were skillled in the "dark arts" which Ganondorf seems to be as well.
"According to Lanayru, powerful interlopers, skilled in the dark arts, rose up and tried to establish dominion over the Sacred Realm using the power of the Fused Shadows."
So, "rising up" probably implies that they were equals with everybody like the Goddesses made people in the begging, but their developed skills in dark arts allowed them to cause chaos. The Sheikah, however, were helpful to the Hylians. They were sworn protectors of the Royal Family. They were not a wicked people. They have no use in Termina, so why would they be a tribe that used Majora's Mask?
In any case, the Sheikah are not an evil race. It's unlikely that they'd try to establish dominion over the Sacred Realm, especially since they were aids to the Hylians. They might have been "shadow folk" but "shadow" and "dark" are now always synonymous. "Shadow Folk" pretty much refers to them living in the Hylians' shadows, doing things behind the scenes in mysterious ways. They are not "dark". And they are surely not from Termina.