This is a difficult question to answer for myself. I find that the wide majority of Zelda games tend to at least incorporate a decent sliver of story into the overall lore. If I had to reach a final answer, I would throw in my two favorites of the series as examples of lore-rich games: A Link to the Past and The Wind Waker.
From what I know about ALTTP at the time of its release, it sort of acted as a prequel for the first two games of the series, establishing early lore though the Sealing War and creating an early framework for an expansive Hyrule in some regards. It also introduced The Master Sword, a staple in the series for numerous games. The story itself, while not strong by today's standards, was simple and coherent enough to be an enjoyable experience. As far as story impact goes, ALTTP's legacy carries on through the Oracle games and LA, which continue the adventures of our friend Link, although these games don't necessarily add to the greater lore as far as I know. (I definitely need to play them one of these days.) ALBW has a more direct link with ALTTP in the sense that it adds more story to the Hyrule in their shared timeline. A new Link in the same world, but with new twists and turns to boot! I never knew what to make of the whole "Hero Defeated" timeline prospect, but after observing the overarching story that ALTTP begins, I find the story on this timeline to be rather intriguing.
WW is an interesting look into Hyrule after the events of the iconic OOT. The story takes a somewhat dark turn that contrasts the lively ambiance of the overall game. It truly sets in the idea that there is unfinished business in Hyrule that must be tied up. Not only does the game tie nicely to the Hyrule we once knew, but is sure to continue the story by embracing a Hyrule we would eventually come to love. The game uses the previous lore to advance a new story that adds to a 'new' lore. The games following WW do just this: they tell the story of a new Hyrule. The stories of New Hyrule don't necessarily add to the overall lore, instead maintaining focus on its own lore. Examples of this include the addition of new villains and legends surrounding them.
It's a tough call to pick a game and/or set of games with the most influential series lore because each has something to give to the overarching stories, no matter the triviality.
