Stone Tower gave me trouble when I was younger. Yeah the Great Temple in the orginal LoZ and AoL were the hardest for me. I think there was a secret to the one in AoL, always go right, or was it left?
left through the first room then right all the way down. once you fall through the shoot on the disappearing bridge, hang a left to punch though the blocks in the floor.
The City in the Sky is probably the dungeon I languished in for the longest, because it was always confusing and I think I just really sucked with aiming clawshots fast enough. It was super long, super annoying to navigate and figure out exactly where the hell I was supposed to go next, and just all around tedious. Not to mention that I disliked the music, the Oocca, and basically everything about the atmosphere. Honourable mentions go to the Great Bay Temple and TP's Lakebed Temple, but surprisingly, not OoT's Water Temple, which took some time but actually didn't give me a lot of trouble.