Its not necessarily the hardest thing ever, just up there in the ranks of hardest temple in OoT. Think about it, the forest and shadow temples were pretty simple. The spirit temple was easy too, just lasted longer. Ganon's Castle wasn't hard, just more challenging when it came to actually doing the things you had to do to progress. The fire temple at times could be confusing and difficult as well, but not near as much back-tracking as was required in the water temple.
The water temple itself had a lot you had to search for, if you didn't find a route that was easy to remember and that you could go through each time you beat it. Personally, I don't play OoT too much these days, and if I were to play through it, I know that the water temple would give me the most trouble.