I do have a theory about why it scares other people, though. I think it's mostly the spiders, skeletons, and ghosts, perhaps the cobwebs and the feeling that you're in an abandoned, haunted mansion.
It's nothing compared to the Shadow Temple, though. The Shadow Temple was truly frightening.
Yes, the mansion feeling is something unique to the Forest Temple. It feels more like a normal building that really any other dungeon of the game, and it becomes more disturbing because of that. I would however argue that the Shadow Temple only managed to be roughly as scary as the Forest Temple, just that they're scary in very different ways.
i think it was because it sounded empty and made you feel isolated...
This is a part of it. The Forest Temple, aside from maybe the Spirit Track, is the closest to being at the "end of the world". It's at the fringe, and it feels and IS isolated. The dungeon is quiet and empty. It is also the first dungeon that you do upon becoming Adult Link, meaning it's the first step in your quest to topple Ganondorf's reign, and that was a scary time in Ocarina of Time, when you were up against a corrupted kingdom. The Forest Temple had many things that returned it to the feeling of
classic dungeons, and that includes a sense of danger that the Child Link dungeons just didn't have.
Combine all these together with the Forest Temple's surreal nature and unique M.C. Escher-like feel, and it was terrifying in its own way.
The music really just complemented this nature, so although it was eerie in its own right, I think it was just a component to what made the Forest Temple frightening.