It is possible to have, Housecarl, wife, steward, two children, Armoured huskie and a frostbite spider. When you arrive with a follower and pet it becomes a small army to protect your home.
And you can still have a follower (or two) and an extra horse if you travel with one not from one of the houses. One of my characters travels with Shadowmere when he is not trying to dump her at Lakeview Manor.