Bombchu bowling alley in MM. I think that one was the worst.
This game is more based on luck than on aiming skills and timing.
Plus, if I remember it right, you have to play it at least 5 times to get all the available prizes.
Of course, you'll fail when they got the heartcontainer or the bomb bag as a prize.
I agree that the Deku Shrub playground in MM was horrible too.
It's not even that hard and the first time it's quite fun. When replaying this game however, it's too slow for a timed game.
Digging in ALttP was fairy random and wouldn't guarantee a win, but at least it's fun to do.
The chest game however, is just as random, but more boring and expensive, so it usually takes longer to get.
Luckily, there are also a bunch of good minigames that I like a lot. Mainly all the shooting galleries, especially the harder ones like the Swamp Shooting Gallery in MM and the horseback archery in OoT. Why? You are in control!
That reminds me, I didn't really like the Goron Rolling Race in MM either. Sometimes you win by great lenghts, other times, you have to try over and over. I never understood races like this. It hasn't got that much to do with skill somehow.