I voted for the Forest temple because I'm sick of complaining about the Water Temple and its infamous key, which is really the only reason it was ever difficult in the first place. Otherwise it's just sort of tedious raising and lowering water levels and switching boots constantly.
The Forest temple was hard for me because it was a little like transitioning from elementary school (The first three temples) to middle school (the last 6). The first three temples are more linear and you have a lot more guidance, in addition to just being easier, as they should be. You don't even have to get boss keys.
Then you go to the Forest temple, your first day of middle school, you get lost in the hallways, trying to figure out which hallway you have to go down to get a key. You have to get a boss key for the first time, you have your first timed block puzzle, the layout isn't linear, the puzzles in general are harder, and the boss is harder than anything you faced as young Link.
The Forest temple was hard simply because it steps up the difficulty from what you experienced as a kid. Please excuse my cheesy grade school metaphor.