No temple or dungeon in Ocarina of Time really gave me any trouble, not even the first time I played it. I think the temple that gave me the worst trouble would be the Forest Temple, which probably sound very stupid. I repeatedly got lost in the Forest Temple, because I didn't know what to do...