OoT/MM: The Gerudo lives in Gerudo's Fortress.
TP: The Gerudo Desert exits, however without any Gerudos.
FSA: The Gerudo lives in the Desert of Doubt, but there is no record of Gerudo Desert.
So, by assuming that there is only one Gerudo tribe in Hyrule (for there is one in Termina) and that the Desert of Dobt is not Gerudo Desert, then the most plausible conclusion is that the Gerudo moved to the Desert of Doubt some time before TP.
This might have a connection to the fact that Ganondorf, their leader. was put on trial and about to be executed by the Hylian government. (Interestingly, FSA Ganondorf was only given the title of "guardian" whereas OoT Ganonodorf was the king.)
/Blue Window