I'd say Navi. Joey wasn't really annoying because, to me at least, he was just another random trainer. My interaction with him went something like:
"My ratata is in the top percentage!"
Okay, lol, go typhlosion.
Navi just plain annoys me. Yes, she helped a bit when I first played the game, but...