The most difficult Elite 4 for me is the Elite 4 from HeartGold, I seem to remember having trouble last time I tried to play through it. But then again, I did take in really low levels in for what was going to be my second run through so the fact that I was some 15 to 20 levels lower with some pokemon didn't really help, particularly thanks to Lance's Gyarados and Dragonite being the perfect pair of pokemon to take down my team thanks to the lack of electric and ice-type moves.
To be honest though, the Elite 4 in most of the other games has never really been that hard, I think the most troubling one after that might have been the one in Sapphire.