As others have said, Hunters will occasionally spawn with horses. You'll occasionally see a nobleman on horseback being escorted by a Legion soldier. Last night I also saw a hostile, unnamed Dunmer spawn near the Morrowind border (heh, fitting) on horseback. He then promptly jumped off said horse to attack me.
...So I killed him and "borrowed" his horse for the trip back to Riften.

I wish when I pointed my follower toward a horse that they would ride it instead of attacking it. I hate that I can't buy a horse for my followers to ride.
A thousand times this. I've quit using my horse very often and started mostly tromping around on foot just because I feel really ridiculous, galloping around on horseback...then stopping every so often to wait for poor Erik the Slayer to sprint five miles to catch up. Having a dog follower trailing behind your horse is one thing, but you being on horseback while your NPC companion isn't just feels wrong.