I don't like the Goron Dancing. That is the most annoying thing in either game. Too hard, even for Zelda experts.
Good gracious this was the absolute worst part of Oracle of Ages. I could not stand this dancing. It always takes me at least 8-9 tries to get by. What I hate about it, it sometimes is just luck, since at times, the tunes that they play for you are quite easy, and then other times there are like 6-7 notes you have to play, and the timing is rediculous to master.
In Oracle of Seasons, I don't know if there is one thing in particular that annoyed me. The game was a lot harder than Ages in my opinion, but I think overall, it wasn't that annoying. The flying cucco that Judai mentioned was a tad annoying.
One thing in both games that I found ridiculously annoying... is getting all the heart pieces. As a completest, I can't stand not having everything in the game. With the Oracles, you needed to get one piece of heart from planting Gasha Seeds, and another from bumping into Maple. I absolutely hated this because there was way too much luck involved in this, especially the Gasha Seed one. Annoying!