Ganondorf said:
When the sun rose into the sky, a burning
wind punished my lands, searing the world.
And when the moon climbed into the dark
of night, a frigid gale pierced our homes.
No matter when it came, the wind carried
the same thing... Death.
Although I'm pretty sure now that he could just be referring to the suffering of the Gerudo people in general, I must have subconsciously associated the word death, in reference to his homeland, to the death of the Gerudo. I thought he was using the word "wind" in reference to fate, as in the phrase "the wind carried," which follows the wind imagery set in place by the rest of the game. I naturally assumed that, in addition to emphasizing the harsh climate, that this was a metaphor for the death of the Gerudo, or a foreshadowing of it. He doesn't outright state that the Gerudo died out, but it definitely seems implied due to the narrative that he was taking with that speech.
Not to mention that a valley is a terrible place to be when a flood hits. I think this does a nice job of contrasting how the other peoples of Hyrule fled to the mountaintops to survive the flood, where the Gerudo have no mountains to climb atop, which further fits in with Ganondorf's whole narrative of the Gerudo being impoverished and overburdened.
I may not be taking his speech as entirely literal, but I don't think that's the way it's meant to be interpreted. In my opinion, it seems too profound and eloquent to mean only what he says. I may be giving the story too much credit, and I'm perfectly vulnerable to the "it's just a video game" argument as a counter, but I think this is a rare exception.
Again, the flood isn't specifically mentioned, but it seems fitting, granted that there have been only two or three Gerudo-like characters in games confirmed to follow TWW on the time line. Again, even if it was not the flood that killed the Gerudo, they cannot reproduce within their society, because Ganondorf was the only remaining male, and he is killed or at least incapacitated at the end of TWW.