Horses are like having one more companion, which means they seem like a good idea, but are mostly a nuisance. They run into that crowd of bandits you want to fireball, or alert that troll you are trying to sneak up on, etc. They either charge into a fight and quickly get themselves killed, or they run off getting themselves lost in the woods forcing you walk the rest of the way (a lot of good that 1000 gold horse did ya). Dragons kill horses so frequently that they might as well give dragons the ability to pick your pocket for 1000 gold in the middle of battle. Plus, getting on and off them becomes so tedious that you stop investigating things and start skipping content. They are like wet tissue paper when you fall off a ledge that your toon would have no trouble surviving.
Horses are basically good for two things. Climbing mountains is so stupid easy with horses that they might as well just give us back jump and fly spells (Beth, I hate you for taking my jump spells). But primarily, horses are only good for hauling my sh!t around (just like mules...err, I mean "companions"). The caveat to this is that making money is easy through enchanting daggers and clothes, or through alchemy. So you don't need to hauling every last iron warhammer out of a cave anyway. And if you just want one to climb a mountain, then just steal one, as getting down the mountain is easier on foot anyway.
Best and worst horse, since if you decide you don't want a horse following you to your fast travels, you cannot ever get rid of him since he's invincible.
What??? Why would you not want a invincible horse? My biggest complaint about horses is how idiotically suicidal and fragile they are.
The TL;DR version:
The only case I could see myself using a horse is if I was playing a hack'n'slash, "business in the front" kind of character that just gets right in a dragon's grill and hacks away. But for thieves, mages, alchemists, enchanters, etc., don't waste the money.