I agree, they need to hop on with you, or pull out their own horse, something...
Is it just me, or can you also not tell them what to equip etc?
Otherwise I find that if I just ride my horse to where I need to go, and wait a good minute or two, the follower will catch up as long as I'm standing on a road, or a legitimate surface an npc can "see" in their pathing...
Like if I'm on the side of a cliff, it breaks the followers pathing, but as soon as I return to a road or meadow (w/e) they find me..