I think when you have company, speaking a language they don't understand around them is rude, because you are excluding them from the conversation by default. I don't appreciate it when I'm with friends from other countries and they start having exchanges in their language with each other all of a sudden and doesn't including me. Or when I'm in a chatroom and people move the conversation over to a language I don't understand. When that happens I switch to Norwegian just to make a point.
I think including people in a conversation is important, if there are people speaking different languages, at least then make attempts to translate between people, let them be part of it. It's not all that hard to do and it really changes the dynamic from a person feeling left out and vulnerable to everyone feeling that they are part of the conversation.
I'm Norwegian myself, one of my flatmates is French and when he had his parents over, we communicated through switching back and forth between Norwegian, English and French. Me and my flatmate in Norwegian, him and his parents in French and me and his parents in English (my flatmate's English is not all that strong, and my French is so poor I can't even have a basic conversation in it). It worked because one part would always translate for the others what we were all saying, like my flatmate would explaing to me in Norwegian, or his parents would explain in English, he would explain to them in French, etc. so everyone got to take part in the same conversation.