I'm also one of the few people that didn't find the Water Temple too hard. You just have to study the map very well and be able to think in 3D to find out where the passages are that connect the floors. And because it's a Zelda game, you just have to go to all possible places within a dungeon. If you keep that in mind then the Water Temple is actually quite easy. There are very few enemies and those which are there don't do considerable damage, I'd say it's almost impossible to die in the Water Temple. Also the boss is one of the easiest one of all the adult temple bosses.
I think that the Forest Temple is the hardest one. It has some fairly complicated puzzles in it and if you miss a key there then this means a lot of backtracking, even more so than in the Water Temple (I'm thinking of that cleverly hidden key in the very first room - I only found it much later after getting map+compass). The enemies in there are quite tough too, I didn't have a fairy, no Biggoron's Sword and only 7 hearts the first time and I totally failed against the Stalfoses, especially where you have to fight two of them at once. Plus the boss was rather difficult too.
In Master Quest it's definitely the Spirit Temple, oh my God, this place almost makes the Great Temple from AoL look like a holiday resort... I don't know how many times I had to change from adult to child Link and vice versa until I finally figured that place out.