The only reason you can believe this is because Zoras weren't there. The barrier around Hyrule Castle was made by Ganon, and it took him a while to regain power. The Zoras could have entered anywhere in Hyrule besides the castle though, even if Ganon hadn't put a barrier there...
Hmm...Let's ponder this question carefully.
Well, there is a giant ocean engulfing the near entirety of Hyrule...
Oh! And Zoras can swim!
This leads me to my knowledgeable and probably accurate assumption as to why Zoras changed into Ritos.
Hyrule was sealed away in the depths of the ocean by...