I agree with Austin in saying that the Shadow Temple was the most difficult of the bunch on my first playthrough of the game. They layout of that place was confusing, and being quite young, I was pretty frightened while doing it. I remember not wanting to do it for the longest time, simply because I was scared of it. However, I finally did get around to doing it, and soon found out that it wasn't really that difficult. It was mostly just very intimidating, as was stated above. Once I got over the intimidation factor, the dungeon wasn't that much of an issue. The enemies in it were still pretty tough though, and even are to this day.
I didn't have to much of an issue with the Water Temple on my first playthrough. It was mildly confusing, but I got the hang of it pretty quickly. I did miss that one key the first time like pretty much everything else, but I didn't have to much of an issue going back and finding it. The boss of the Water Temple was probably the most annoying to me, primarily because of that really annoying room with the spikes right before it, and because I didn't know any of the good strategies to beat the boss on my first run through. Really, the Water Temple was annoying, not difficult.
As for the Spirit Temple, which is often cited as one of the other most difficult dungeons in the game, I didn't have any troubles with this either. Yes, the enemies were pretty tough, but nothing that I couldn't handle. Even fighting Iron Knuckles as a child wasn't that difficult, as they were really slow. Having to get in close to them could be a problem, but really it wasn't that difficult. I don't really remember any issues that I had with any puzzles in that dungeon either. I remember it being challenging, but nothing to difficult or annoying like the previous two dungeons. It was fun, however.