Ocarina of Time isn't actually my favorite game right now, but it has been, and it will most certainly become my favorite game again sometime in the future. It really is a great game.
If you look at the content of Ocarina of Time you can see that the game contains all of the most important "Zelda stuff" so to speak. You can do almost all things that is Zeldaish, and I think that is quite uncommon amongst the other Zelda titles. You see, in Ocarina of Time:
You can play as both a child and an adult.
You can travel through time.
You will meet all of the classic races such as: Kokiri, Goron, Zora, Gerudo and Sheikah.
You can ride Epona.
You shall battle agains both Ganondorf and Ganon.
Everything takes place in Hyrule.
You can play music.
Navi is Link's partner.
(The last part might not be a real reason though...)
Also, the game is not as simple as TP or WW, but still not as difficult as LoZ or AoL. Ocarina of Time has a medium difficulty, and I think that is the best for most of the Zelda fans, and I think that's a major reason.