There's a difference between "challenging" and "difficult". Challenging means "testing one's ability and endurance". Difficult means "not easily or readily done". SS is definitely the most challenging, as it tests all kinds of abilities to the max. It is definitely not the most difficult, though. That title belongs to Zelda II.
This isn't a bad thing, though. Games don't have to be insanely difficult in order to be good. I see way too many people complaining that Zelda is "too easy" these days. People, Zelda is a casual series. It may have the most hardcore of hardcore fans, but it's still a casual series at heart. Of course it's easy. If it were really difficult, not everyone could beat its games. Most seasoned veterans also know what to expect from the games, so they know what to look for in the puzzles and enemies and what not. The games are going to be easier to them due to this. SS, imo, actually had the ideal difficulty level for Zelda. Not too hard, not too easy. It was at a just right level where there was never a moment that felt boring nor a moment that felt frustrating due to the difficulty level, vastly due to its excellent pacing. Just grow up and quit complaining. Play the games and enjoy them.