I don't use them often but for role-playing reasons I have tried a couple times. The problems I have are:
1) Force third person. Sorry but I just don't like it.
^ This.Third person is nice and all, and I use it on occasion, but I'd also like the option to be in first person while on horseback. Why it's not available is one of those great "what were they thinking?

" moments of Skyrim.
2) Can't do anything while riding. Dismounting all the time.
Indeed. My character makes frequent use of alchemy, so one of the major reasons I don't use a horse is because I would be stopping literally every five feet to dismount and gather alchemy ingredients. No mounted combat is one thing, but not being able to reach over and pick a flower, or to stop and talk to someone? Really? Major step backwards from Oblivion, which is saying something in my book.
Other reasons I don't use horses much include: no mounted combat, and no horses for followers. No horses for followers is probably the biggest one. Oblivion modders had this figured out ages ago, and most of the big companion mods (Vilja springs to mind) have horses. Heck, you can even tell Vilja to ride her horse even when you're on foot!
As for the mounted combat...true, Oblivion didn't have this either. On the other hand, there was a mod for Morrowind (Pegas Horse Ranch) which not only added horses, but it added mounted combat. And horses for followers. Granted these features were rather clunky, given engine and script limitations, but hey--
at least they were there.
It's really

-worthy for me to see mods for Morrowind and Oblivion that managed to include features vanilla Skyrim is blatantly lacking. We won't get into Morrowind and Oblivion companion features vs. Skyrim's vanilla followers.
Bethesda, why you no use teleport rings?!