Here's mu problem, the only returning character is Ganondorf, Link only returns for a few direct sequels, otherwise it's a different character. In other words, there isn't much connections between the games except that "this is the history of Hyrule, not the characters that you know and love".
That's because you're only establishing connections with characters. If you take in the events, the land, even things like reused items, the connection of the story really comes into play. This is difficult to see in Zelda because it doesn't do a good job with continuity. It constantly changes not only the characters, but the land, and even the very history of the series. This is why fans make a big deal out of these changes because we're losing immersion. Of course, not every game's story needs this kind of continuity to make the player feel immersed, but Zelda has set itself as an action-
adventure game which means its story will be better should it hold a bit more traditional literature concepts.
I'm not saying people will hate Zelda games if they don't have proper continuity (as they don't, and people love them). I saying fans make a big deal out of it because we want the best from our games and this is something that could easily be improved upon. We don't need the same characters, we just need the same history.