The way I see it, there is a big difference between 'legendary', 'best' and 'favorite'. Legendary refers to the revolutionary aspect of the game, best refers the most well made and favorite refers to personal preference. Now, I'm just going to cancel out the entire idea of 'best' because there's just too many ways to look at that. I don't believe in 'best', especially in cases like this. Onto the topic at hand, while I do prefer Majora's Mask and A Link to the Past over Ocarina of Time while also putting Skyward Sword on its level as my 'favorite', as far as I'm concerned, Ocarina of Time still has that legendary status for its innovation, its revolutionary ideas. Sure, we could sit here and argue that if it wouldn't have been Ocarina of Time, it would have been something else, but the point is, it is Ocarina of Time.
Ocarina of Time took a huge leap forward from its predecessor and in the gaming world in general, which I feel is really the key idea behind deciding whether its legendary. Before Ocarina of Time, the most recent Zelda game had had dungeon crawling across a 2D Map with a pixelated Link. It didn't have all these open world, 3-D details, it was just a what it was. Then you got Ocarina of Time which changed things entirely. It was an entire large 3-D World with many different innovations. No longer did you just pull out a musical instrument and it automatically play you a song, in Ocarina of Time, you actually had to learn the song and play it. This feature has been used in every Zelda game that I've played since then. While I'm not really into the way certain games have done it, it's still a feature that is used and has been used since Ocarina of Time.
Ocarina of Time used the ideas from A Link to the Past and put them in 3-D. Why isn't A Link to the Past considered the most legendary? Well, really it is still a pretty legendary game. The leap forward from Adventure of Link to A Link to the Past is just lesser than the leap between A Link to the Past/Link's Awakening and Ocarina of Time. You not only got to control the way your character moved in three dimensions in a smooth way, but you also got to ride a horse. You got to ride it in three dimension and control its movements. This way of riding a horse has been used in several Zelda games sense then. Let's also consider the games set up. In my opinion, its story is not as good as Majora's Mask and a couple of other titles, but what it does have is variety. It has more than just good gameplay and good story, it appeals to a very wide audience. Where other games in the series tried to put a focus more on certain things, Ocarina of Time found a general appeal to the majority of gamers of that time.
If you play games today, you will find that some of these ideas are used in games, even outside of the Zelda series. Look at the Darksiders series for a second and the way they use their horse. It's pretty much a carbon copy of the way Ocarina of Time used Epona, except that in Darksiders the horse arises from the ground instead of running to you. Zelda also epitomized the idea for 3-D dungeons and puzzles with Ocarina of Time. Those ideas are used in many games to this day. Think about Guitar Hero for a second. While I don't know if their inspiration came from Zelda or not, whatever did spark their idea, was probably from something that originated with Ocarina of Time's ocarina playing system. It also goes without saying that Ocarina of Time gave us a big leap forward into the 3-D world.
As I said before, I prefer Majora's Mask over Ocarina of Time personally and feel the way they used time, side quests and the way they combined them is underrated. However, Ocarina of Time really just gave us a leap into the future, to the realm of 3-D. Its features are used in many games today, not just in the Zelda series, but in many Adventure games. Can its legendary status be taken? Perhaps, but it's going to take a whole heck of a lot to do that. Who knows, maybe the Zelda series could be the first one to send us into a virtual world. That sounds like it would be pretty revolutionary to me.