I would have to say Phantom Ganon.
I have a few reasons of why he was to hard and why the other bosses were easy:
When playing volleyball with Phantom Ganon, it is much harder than with Ganondorf. Because of this and a few other things, I lost five lives when fighting with Phantom Ganon.
Let's talk about the other temple bosses, as well as Dark Link:
Volvagia was to easy, I only lost one life with him/her. He just threw you rocks and it was hard to get hit by its fire.
Morpha isn't to hard, you just have to be carefully with his tentacles. It's annoying though.
Bongo Bongo itself is an easy-medium, you just have to hit his hands with arrows, then watch him with the Lens of Truth to hit his eye. The problem is the place where you fight him, it's to small.
Twinrova is the longest, but one of the easiest bosses in my opinion. You just have to reflect one of the sisters's magic to the other one, then absorb three powers with your mirror shield and hit her enough times to kill her.
Dark Link itself isn't to easy, but you can kill it with many weapons.
If you just buy one or two magic potions before oyu fight him, then use Din's Fire all the time you can kill him easily.
Well, that's my opinion!