I always leave the cartridge in a handheld, otherwise I'd be concerned about dust, or even accidentally putting too much pressure in the empty slot.
For consoles, I don't like to leave games in, even if I know I'm likely to return to the same game soon. I try to take as much care as possible...