I wasn't really scared by the Forest Temple. The only things in the Forest Temple that scared me were the Wallmasters, when they came down unexpectedly and grabbed me.
I do have a theory about why it scares other people, though. I think it's mostly the spiders, skeletons, and ghosts, perhaps...