There are always certain games that pull you in when you play them for the first time. Whether it's the gameplay (Zelda, Mario), the great atmosphere (Metroid), or the amazing story (Fire Emblem, lots of JRPGs), something about it just clicks. For lots of gamers, the N64 era was great for these types of games: Mario 64 is seen as a revolutionary game, and as this thread indicates, many people see OoT as the best Zelda, or best overall, game of all time. As others above me have said, part of this has to do with the age at which we played OoT: the world seemed much more expansive and full of wonder to us as kids. However, the impressions you get playing a game the first time tend to stick with you, so (for me, anyways) lots of us likely get the same feelings from it now as we did then. Back then, there were three main things pulling me into it: the sense of wonder and discovery when exploring; the enjoyable and challenging-but-not-impossibly-difficult gameplay; and finally, all of the extra things you could do outside of the main story.
The exploration in the game was fantastic. Each area was unique, with great, memorable music to go along with it. Discovering a new place was exciting, and most were big enough that you could spend a little bit of time checking out all of the nooks and crannies. As Child Link, despite the various troubles each place was undergoing, there was a sense of liveliness at each area as well: people bustling around all over the place, normal life going on. This provided a great contrast to the change that comes over everything when you become Adult Link. Suddenly, everything is entirely different. There are few people, and everything has a dark edge to it. Not only did this make each area feel totally new when you revisited them, it also gave you a really good reason to want to defeat Ganondorf and get everything back the way it was. I personally loved the bittersweet feeling of the changed Hyrule.
To compliment this dark new Hyrule, the difficulty also ramped up a bit here. The first dungeons weren't too hard, all things considered, and they smoothly lead you into the game. These later ones, however, have tougher enemies, are really confusing in parts, and are a lot longer. You also aren't given as much direction as before; while there definitely are enough hints that it's hard to get stuck, it also means you have to try a little harder to figure out where to go next (I remember it took me a while to figure out how to get into the Forest and Fire Temples). This has a lot to do with the enjoyment of the game, because it lends a sense of accomplishment when you make progress, and it also lengthens the game. This is one aspect that makes replays not as enjoyable, because each time you go through it you start remembering more and more of the puzzles and boss/enemy patterns. I know you can make it more challenging for yourself if you want (and I have done the 3-heart challenge), and I know that it does add something to the replay value, but it's not quite the same. I've practically memorized the main quest, and so it feels a lot less rewarding to get a new item, or complete a dungeon.
That being said, the side-quests, on the other hand, have remained quite fresh for me over the years. Since you don't have to do all of them when you play the game, they aren't all entirely familiar to me. They also contributed a lot to the game when I played it the first time. The idea that they were extra things, that you could decide whether or not to do them, and when, added to the sense of exploration and freedom (even if the game was largely linear). There were quite a lot of them, too, between the heart pieces, golden skulltulas, and other various quests. Lots of the rewards for those other quests were pretty neat: the magic spells, the arrows, and, best of all, the Biggoron Sword. The extra little tidbits of information you learned about Hyrule and its people while doing these quests, while perhaps not too deep, also made them interesting. Often I would look forward to completing a dungeon just for the extra side-quests I would be able to do before going further with the main story.
These are the three main reasons I enjoyed, and continue to enjoy, Ocarina of Time. I know people often say that nostalgia is a huge factor as well, but I would argue that that's just another way of saying that your own childhood impressions of the game have stuck with you, so that you still get a little bit of that wonder when you replay it. It may never be quite so awe-inspiring as when you first walked into the Deku Tree, struck out across Hyrule Field, or got your first glimpse of the ruined Hyrule Town of Adult Link's world--but there will always be something that will inevitably draw you back to it every few years or so. In my personal experience that has been true, and I'm sure it will continue to be.
