As a general rule, I don't ever really have too much trouble with the dungeons, so it's kind of hard to think of any one instance that stuck out as being particularly difficult, but I suppose when I really think about it, of the games I've played, my first times through the Great Bay Temple and Stone Tower Temple were pretty confusing. The Great Bay Temple frustrated me a bit until I got down the layout of the dungeon securely in my head, and after that, all the problems I was having went away, since... it helps to know where you're going, obviously. With the Stone Tower Temple, I just remember a couple instances where it took me a lot longer than average to figure something out, but that just made me love the dungeon, honestly, instead of making me annoyed with it. I vaguely remember having some issues with the City in the Sky, but my frustration may have been caused by the fact that I thought the dungeon had kind of a choppy flow to it.
That's really all I can think of now; I hope to encounter more difficult dungeons in the games I've yet to play, when I get to them. :>