Well, this is a little theory of mine, but I'm wondering if Gerudo women sometimes engage in homosexual intercourse. If only one male is born per generation, the women might be lacking sexual partners. Of course, we don't know if Gerudo or monogamous or polygamous.
The same goes for the males...