Yes, but he was also able to defeat Ganon. Magic sword or not (and I don't think you even use the sword in that battle) that's pretty good.
Also, use of a sword is one thing, but look at the number of different items he had to be able to use. The flute, the bow and arrow, the boomerang the bombs...