I was actually hoping that with the rerelease of OoT on 3DS, Nintendo would add back all the extra story that was said to be removed to fit the game into the N64 cartridge. A few developers over the years said that they had to remove a lot of the story and character phrases because it did not all fit. So things were brought down to the most important game and quest related events and messages relevant to the story. I hoped that they would add back the extra quotes lost on the cutting room floor and it would clear up a few of these other questionable things that happen in and around Hyrule. Gerudo and everything about them was definitely one of those things I wished to learn some more of. We actually know so little about them, especially since they have only appeared in three games and have not made any recent appearances at all.
We mostly have two theories to go by in the game. First there is the Ganondorf, single male of the tribe thing. Which causes many to consider him the father of the next generation of Gerudos. And then there is the gossip stone which says Gerudo go to Hyrule for boyfriends. Again Hyrule being a magical place, certain races might have a few special magical properties to help them along that makes little to no sense science wise. It might be that any child of a Gerudo is a full blood Gerudo regardless of what the father was. I have seen something like that in fantasy stories before. While also their gene pool does not get any worse when there is a single father for a generation as well. It could be some kind of magic they are using, or divine intervention on part of their goddess, we do not know because it has still never been explained by Nintendo.