After playing Ocarina of Time for at least 10 years now, I can say with confidence that nothing strikes me as hard anymore, especially the Shadow Temple.
In my opinion, the Shadow Temple is of the easier side of the difficulty spectrum of the dungeons in Ocarina of Time. If you know what you're doing, you can blast through that temple amazingly fast and with relative ease. However, I can see how it can generate problems for people that haven't played it as much as someone like me has.
The fact that some of the enemies, and traps are invisible, can make it very difficult to get through the temple. Also, the Hover Boots, if not practiced enough, can be a real pain to get used to. Not too many people know the tricks to making them work right.
So, I agree with you that it isn't difficult, but I can see how it could be to some people.