I've always seen it as a missed opportunity. Here's a character who could've been fleshed out with a personality and quirks and motivations and contributing to the plot, but instead they stand around gaping like a hole in a fence the whole time.
This idea that a silent protagonist helps with immersion is bupkis. Of course, characters like Freeman, Jack, Corvo, et. al. don't have to talk, but being silent does nothing to help foster immersion. I'm not a nuclear physicist, space marine, plane crash survivor or royal bodyguard in the first place.
Let's compare a silent protagonist to Garrett from the Thief games. Garrett is seldom seen (even by the player, being a first person game and all) but often heard, at least in between-mission briefings and cutscenes where his voice is the primary vehicle that drives the plot along. Garrett is voiced brilliantly, and his voice is the major defining aspect of his character. Garrett is a thief who spends the entire game sneaking around stealing things unseen, but his vocal expression really brings him to life. Imagine how dull the game's story would be if he never said anything.
The only games where the silent protagonist schlickt is done effectively are the Myst games, because in those games you are the protagonist. In the contest of the game, it is you who are exploring these mysterious islands and the game goes to great lengths to never forego that pretense by assuming anything about the personal aspects of the player that aren't true. Whenever the player is directly referred to, it is through ambiguous terms such as "stranger" or "my friend."
Of course, players have no way of expressing themselves through vocals in the game. Whatever actions the player has is limited to environmental interactions. There are no dialog wheels or trees. Any story decisions are determined by what actions you take.
Still, even here it is rather jarring to assume that on the few occasions you do have direct face time with another character (these moments are rare in the series) the other person is just going to be allowed to stand around and monologue when you - the player - reasonably have about a million pertinent questions for them.