There are two gyms (Or groups), in my view, that really take the cake for me in this case:
Elesa, of Nimbasa City: It took me a good six tries to take her down. It was her Emolga that really did the damage, with its Volt Switch and all, even my Boldore at the time couldn't take Elesa.
Then, there's pretty much any Dragon Type Gym, which I feel has been overused in the series for some reason. It's not even like they're overkill, but it's a big pain for me to have to get some ice-type pokemon that I could barely train and what not at the current point, or just try and find some TM's to boost my water type pokemon capabilites. Nonetheless, even though in general, they are difficult to face, which is true, but the Dragon Gyms are just major pains in the behind for me.