I have read this up and down, ALL the comments. I do believe that the Gerudo are in fact the Twili. To much evidence is pointed towards that. I'm sure Nintendo made it clear, but as us Zelda fans are we analyze to much based off of past links. Mind you every Zelda game is linked in some way but have the major individuality. There are plenty of loops holes I assure you, if Nintendo tried to base all the games off of each other then that would constrict their creativity. They said it themselves, also the time line has proved to be flawed.
Enough about that lets shed some light on the subject, Avenged did an awesome job in doing so I couldn't have done better myself.
First off as many have pointed out, the Gerudo don't rely on one male alone to sustain their whole race. The Gossip Stone and several NPC's have made that clear that they look for boyfriends. As we all know the Gerudo don't like men so I assume they wouldn't want to build a life commitment with one. In other words USE them and abuse them, for them that's all they're worth.
As someone stated and a good point at that about the male to female ratio in the twili realm. Well as said good point, but as another said this event takes 100's of years prior giving ample opportunity for many or so men to be born. But there are many ways of figuring this out, or different theories to plug this in. One could have been that some of the Gerudo were pregnant when they entered, or there could have very well have been a race there before then and the emerged with them. Or there could have been a mix among the race as they were banished. All can easily fit in nicely and be plausible but we may not know.
To go with these maternal theories with the Gerudo giving birth, and what not. Here is food for thought. There is a rumor based off of what was said in OOT. About Malon's mom when you wear the Gerudo mask in front of Talon. He was more than delighted to say the mask resembled his wife. It could have mean one of many things, the red hair, it was a female mask and he misses his wife, or he said that and quickly regretted it trying to take it back due to the fact no one in Hyrule like the Gerudo's and he didn't want anyone to know he was tied to a dangerous thief (I'm sure plenty of men were). The point I'm trying to make is that if this theory is true than Gerudo's didn't give birth to all females or look alikes at that. They probably had a strict code in which stated rules and regulations in how a Gerudo should be part of the tribe, assuming appearance being on of them.
Also another thing I want to add is no one stated when the banishment actually took place. We know it was a long time ago, we don't know when. Gerudo could have been banished right before or after Ganondorf was supposedly executed. When you watch the execution clip. Noticed how everything is in place the mirror is up and the giant rock. The Mirror is a powerful artifact something that shouldn't be out in the opening like.
Also think about it the Gerudo had a thing with mirrors. In the spirit temple the whole thing was riddled with mirrors. In fact that is the only way to get through the temple is to use a mirror including your sheild.
Besides all that and the other little tidbits that were added with the other comments. Let's look at some other things, the fact that the Arbitor's Grounds could very well be the Spirit Temple due to the sand and the lady with the snake plastered everywhere.
Here is something else I would like to add, some of you said about the loyalty the Gerudo's had with Ganondorf when in fact they had a huge loyalty, with the exception of Nabooru. Not only that but a lot of them were often brainwashed but Ganondorf, Kotume and Kotake. As well it was the only army Ganondorf had to his disposal, that and the fact this takes place after OOT which OOT is the beginning of the time line.
Mind you, Midna might as well be a Gerudo she does in fact resemble one. The the eyes, the long thin neck, slender waist, orange hair, aquiline nose, though it may not protruded as the Gerudo depicted in OOT did. Put into consideration that the character design has changed in the transition of TP.
And to what Ganondorf said to Midna about "Her People" think about it you can no longer associate yourself with something your no longer tied to. My Great, great, grandfather was from Germany, that doesn't make me German now does it.