I have a whole worldbuilding theory in my head regarding this very thing.
You see, I believe the Gossip Stone outside the Temple of Time in Ocarina of Time - the Gerudo sometimes come into Hyrule to look for boyfriends. Though an all ages/kid's game, this... gives a nice little implication...