I honestly think people over exaggerate the difficulty of MM. Sure, it was harder than OoT, but it was still pathetically easy.
Anyways, I think that AoL was the hardest. After all, it made the combat extremely tough and, though they were much less numerous than in the original, the puzzles were extremely devilish. After all, how was I supposed to know that all of a sudden in one of the Palaces I was supposed to walk through a wall or to find Bagu's house on my very first run of the game with so little knowledge of it?