You can put this in two perspectives; the fysical journey and the emotional journey. Both are different things and would impact Link in different ways. Both are also dangerous in their own ways; collapsing from exhaustion or going insane over mental problems. Which one is worse?
I'd say the hardest fysical journey would be Twilight Princess. First of all, Hyrule has never been bigger. Link has to travel way more than in any Zelda game ever. He also has to survive some of the harshest terrains we have yet seen in a Zelda game. A freezing ice wasteland (in nothing more than his tunic!) or a scorching hot desert. Or what about getting shot into the Sky with a giant cannon? Don't think that will do wonders for your back. The TP bosses are some of the toughest I've seen in Zelda games. Two epic 'final' bosses with Zant and Ganondorf, a giant skeleton, a huge dragon, a enormous seasnake... I mean seriously, the game is only one hour underway and Link already gets knocked out by a dude with a giant bat!
More interesting in my opinion is the emotional journey however. There can be only one Zelda game that takes the title for most emotionally challenging Zelda and that is offcourse Majora's Mask. There are so many reasons why this journey would drive a little boy insane. First of all, you have the weird, trippy looks of it all. Like Link didn't even fall in another dimension but just ate a large portion of magic mushrooms

. When he finally wakes up after losing his one and only friend Epona he is instantly turned in one of the things he had to fight all his life; a Deku Scrub. As a tiny little monster he instantly becomes the outcast of Clock Town. When Link first enters the world of Termina after another acquintance with one crazy individual (Happy Mask Salesman) he finally sees the Moon. Seriously, that face even gave me nightmares when I was little, imagine what it would do to Link. How paranoid would you get if you knew you had only 3 days to live and even save the world? And all of that while being a helpless little Deku Scrub. Knowing that everyone you meet and see will simply die in 3 days?
Then Link recovers the Ocarina and gains the ability to travel back in time. He is able to escape the destruction, but what about all the other people? Link travels further through the land of Termina and comes face to face with more suffering than he ever experienced before. A desperate father falling to his own gruesome death, leaving his own son to save him and his people from freezing to death. Or what about a brave guitarist that dies while trying to save the kidnapped, unborn children (WTF?! Nintendo?) from one of his closest friends? And we haven't even talked about the little girl who's father turned into a monster in the midst of one of the most haunting places ever seen in Zelda. Or the creepy dancing guy that can't stop dancing. Or the falsely accused monkey that would be burned alive. And all of this while that creepy moon is constantly coming closer, the clock ticking away untill everyone's final doom.
Seriously guys, there isn't even a discussion. Majora's Mask would drive every normal person insane.