...I left him at the whiterun stables when I tried buying a new horse for my follower. Not long after that he left.
That may be the issue. I've been told that we can own multiple horses and they return to their stable of origin when you take one of your others. IDK how the game decides which horse goes with you when you Fast Travel, but with all the various bugs I'm betting the game doesn't handle this very well.

I guess you've checked the eery pool from which he originally rises?