As much as I love both games, and as much as I love OoT (see the signature if you have any doubts
), I'm going to have to say that Super Mario 64 was a more essential step. It was the first game to actually have a functioning camera control in 3D (although it would be considered horrible by today's standards), and most 3D games were downright unplayable before that point, merely because of the camera. It really fixed a lot of issues, and one of the best things about it was that you were able to control the camera yourself if it wasn't working out for you. (Which is a smart move for getting critical acclaim by Nintendo, because if you criticize a camera that you're controlling, it's your own fault if it isn't working out
)
The camera wasn't the only thing, either; it managed to create some of the first 3D boss fights (the only earlier ones I can think of are maybe from SNES Starfox), which I imagine required a fairly substantial amount of coding for the time. Another thing as that most 3D games to that point had been on rails, often because of camera difficulties, and Super Mario 64 was one of few games for its time to have a hub world with lots of interactions, and lots of places to explore.
Ocarina of Time certainly pioneered a fair number of things, too, even to the point where it was cited in one review as being "a walking patent office." One of the things it's most recognized for is lock-on, which was a really great way to handle a lot of the issues that came about from having to manage the camera and fight at the same time in 3D. It had a very versatile context-sensitive A-button, and it was one of the first console games to really have hotkeys, in the form of being able to have 3 items out at once; then again, that's mostly due to how many buttons the N64 controller had, but that's beside the point. Before Ocarina of Time, there wasn't really a whole lot of diverse actions you could do without going through a menu very often, but some would argue this was still the case (
...Iron boots...).
It broke a fair amount of graphical ground for its time, even though it seems pretty simplistic by today's standards. Watching some of the end sequences (particularly the final battle) are still graphically impressive, mostly for the level of detail and scope found in that part of the game.
And then there's innovation and strict gameplay design. That was definitely it's biggest success, and I'm sure I don't have to tell you about it or why it was impressive. It just darn well is.
Despite the disproportionate amount of gushing I did over Ocarina of Time, I still maintain that Super Mario 64 was more influential on gaming as a whole, but Ocarina of Time was pretty magnanimous, too.