the only time i remember have difficulty against the e4, was in crystal(i was like 7), it took me like a week to beat them. although it was super fun becuase me and my friend had a bet to see who could beat them first. though the winner was never actually decided for some reason.
The most difficult Elite 4 for me is the Elite 4 from HeartGold, I seem to remember having trouble last time I tried to play through it. But then again, I did take in really low levels in for what was going to be my second run through so the fact that I was some 15 to 20 levels lower with some pokemon didn't really help, particularly thanks to Lance's Gyarados and Dragonite being the perfect pair of pokemon to take down my team thanks to the lack of electric and ice-type moves.
To be honest though, the Elite 4 in most of the other games has never really been that hard, I think the most troubling one after that might have been the one in Sapphire.