Majora's Mask... does feel like a nightmare. Not one that's terrifying at face value, but everything about it always seems just that little bit... off to me, and I think that's what does it once you look a little deeper (and wish you hadn't). Maybe (like MC, FE and RP said) it's because it's not your typical "You must save the world, and you will definitely save the world" scenario most of the other Zelda games have, and so from the moment you turn your console on you know it feels different. Nothing's ever certain and it causes you to question yourself, which halts your progress at times. The odd assortment of instruments they use in the background music and the bright, contrasting colours only add to this sense of off-ness... like a rather vivid nightmare. It is difficult to describe... I've always thought it feels akin to a nightmare where think you've woken up, so you get out of bed and suddenly, all the furniture in your room is stacked on the ceiling, and the sky outside is pink and yellow... and when you look back the walls are starting to melt and the shadows all look darker and almost seem to leer at you... but for a second you don't quite realise you're still dreaming, and there's this detached feeling of confusion and vague panic. "What's going on?" "This isn't possible" "What happened to reality"... It's like that. Surreal, I suppose you could call it, and the game oozes it... from the story to the locations to the situations to the way the majority of NPCs deal so calmly with the coming apocalypse.
The oppressive feeling the constantly leering moon creates is also a big part of it, as people have said XDD
I also think the fact you feel like you can't actually save everybody at once also adds something to it... there is no heroic third option where no one suffers and everyone lives. MM forces you to face it at some point... the fact that you only have so much time, and so you have to prioritize on who you're going to help. Are you going to save the monkey and the Deku Princess, and let all the Gorons starve and freeze, and Mikau die alone on the beach, and Anju give up on her lover? Or are you going to stop the snowstorm at Snowhead, and leave the monkey to be burned to death for a crime he never committed, and let little Pamela face yet another day, alone and trapped in a house in the middle of Ikana Valley with only her half-gibdo father for company? Are you going to spend your three days saving Lulu's eggs, while Romani is abducted by Them and no matter how much her sister pleads she no longer remembers her? Every minute you play, the thought of this looms over you... it's your call, but you always know in order to save the majority of people in the end you're always going to have to leave someone to suffer. And you wonder... because you consciously chose not to help someone, to condemn whole cities of innocents death, can you still truly call Link/yourself a "hero"?
That sort of issue's never really been explored in the other games, and it's like the game has a tighter grip on you... almost daring you to make a mistake you'll feel a twinge of guilt about later (because you wasted time and failed to save someone). Or like the game's snatched something from you, and you're subtley aware you're no longer quite in control of everything that's going on in the game... you can only influence it slightly, and only ever one small area at once.