how man here keep losing their followers?

twice today i've lost j'zargo in quests, the first time after i finished my quest i found him back at the collage of winterhold.

Never. They just take a while to catch up.