I think there's an annoying enemy from every game I've played:
Ocarina of Time: ReDeads/ Gibdo (I am not for one on face-noming), Deku Scrubs (in child from if I had no Deku Shield), Mini Peahats, Shellblades
Phantom Hourglass: Crows, Green and Blue Chu Chus,
Twilight Princess: Chillfos and Freezards. ReDeads were pretty annoying, but they lost their frightening aspect with the amazingness that is the ball and chain.
Skyward Sword: kulltullas especially ones that are on the ground. They attack like ReDeads too, which is not very welcome for me.