actually this is a tough question.
I personnaly like games with a high difficulty.
But yeah, if a game is really really difficult at some point that you replay over and over again, and then you get really frustrated, then it gets annoying. But then again it depends if you enjoy the game or not.