Impa really scared me. In the scene after you fight Scaldera, she just whips around and gives you the evil eye. I felt like she thought of Link as nothing more than a simple child, and she didn't seem to believe he could help Zelda. Impa just looked and acted so powerful and all-knowing in that scene, and the fact that she points out fault in Link means she probably wouldn't have been scared to aggrivate him and take him in a fight. Impa's unwavering confidence would make almost anyone think twice before back-talking to her.
I mean, those Silent Realm gaurdians. That music makes it even worse ._. Although, they add a really cool gimmick to the game. I actually never had a problem with them, never died in any of the challenges.