They will be where you first encountered them. If they are a housecarl, they will be at the appropriate house. If they are a mercenary, they will be at the tavern your first recruited them...same same for everyone else.
However, for those who get confused, a system similar to Mount and Blade Warband, where gossips and travellers could tell you where they are would be handy (but in Warband your dismissed companions/followers used to wander around the countryside, so they could litterally end up anywhere).