Hmm. I'll have to echo everyone else by saying that the Stone Tower Temple was probably the hardest. To be honest, I've never completed it all the way through (not because I was stuck, but because my Gamecube version kept freezing) but I got through most of it, I believe. I didn't find the flip puzzles particularly difficult, the area where I had the most difficulty were those friggin' mirror puzzles. That one room where you have to charge up a mirror with another mirror while black poes come at you in all directions.... that singular room may have been harder for me than the first three temples combined.
As for bosses, I haven't faced Twinmold or Majora, so I'll pick Odolwa. That is, facing him at the beginning of the game with only three hearts. I've never been able to figure out the pattern behind this guy's moves, each time I face him I win through a combination of luck and fairies. I found Gyorg impossibly difficult the first time I faced it, but I didn't get a scratch on my second try. Once I learned how it operated, it was pretty easy to take down.