I often find myself in a game where I am one of five or more players but all of the players but I are on "the other team".

The team balance thing should prevent team switching if there is a more-than-one disparity between the sides and there is no volunteer trying to switch in the other direction.