To be honest, I rank difficulty based on three categories:
1. Is it fair, by which I mean do you know what you did wrong when you die.
2. Is it even beatable, take for example the first arcade units of Pac-Man.
3. Am I still having fun even with the difficulty turned up.
And if I'm not having fun I would say would be difficult. Therefore I would have to choose Scooby-Doo Night of 100 Frights. But the first three Castlevania games are coming in a close second, the only reason why I don't consider them difficult is because I haven't played them as much.