I believe the Forest Temple definitely has its scary moments, however the fact that it is your first full fledged adult dungeon plays a huge part in that. The overall environment has a gloomy atmosphere with the introduction of the four Poes, the creepy Wallmasters, and the chilling music. Despite this, the Shadow Temple is far more dark and frightening..
Let's start with actually getting to the temple.
Listen to this song while reading this post. You enter Kakariko Village and see it's in flames, and under complete disarray. You see Sheik standing at the well, screaming at you to get back, as an evil spirit bursts out from the well. Once you manage to get back to the past, you have to complete Bottom of the Well.
This place is a nightmare, but foreshadows events to come. You are in a massive underground area with obvious blood on the walls, skeletons of dead bodies everywhere, and on top of that, you get to fight
Dead Hand, a demonic creature that will literally bite chunks out of you, if you are unable to escape it's grasp.
After you get the Lens of Truth, you are able to enter the Shadow Temple. Holy ****. This place is ridiculous. ou get to listen to the soundtrack, which is extremely ominous and dark. You venture through rooms of torture chambers, blood stained walls, another Dead Hand, spike traps, guillotines, and a damned shadow ferry. Once you complete this torture-death-chamber of a dungeon, you have to fight Bongo Bongo, a giant shadow beast with a severed head wound as a weakpoint.
Shadow Temple, for me, is definitely more frightening than the Forest Temple. It has everything you want in a scary dungeon, and then some. 11 year old me was definitely happy when I could finally move on and head to the desert.