You mean What Was the Worst Thing that Ever Happened. Since we seem to be talking about Ocarina of Time, I would have to say.....
Getting lost in the Shadow Temple. it took me forever to find a second small key. i had to use a freaking WALKTHROUGH online by nintendocaprisun of youtube.