PhantomTriforce
I am a Person of Interest
Shadow temple, because it's well... the hardest, lol, but also I have a tendency to forget talking to the Scarecrow after becoming an adult, so when I play the Scarecrow song I don't understand right away why Pierre doesn't show up... lol, and if I don't notice that mistake prior to entering the temple, I have to leave and go back in again... very annoying...
Just so you know, the Scarecrow is not required to complete the main quest of the game.
However, on my first playthrough of the game, I found the Forest and Water Temples equally hard (but I voted for the Forest Temple anyways). For the most part, the Water Temple was just annoying, with tricky puzzles, which made is even more obnoxious with all the water levels. The Forest Temple had me stall in a few more places the first time though, and it is a really well-designed temple in my opinion.