Not sure if this has been brought to attention. Whenever there is a follower in a party, there is a way to loose him.
If one travels through area occupied by enemies and go through doors/whatever leading to another area, the follower gets lost. He wants to fight the enemies and does not join player in the new area.