I really didn't like horseback riding in Oblivion. I just felt that control of the horse was unnessesarily difficult and is the reason why I have ignored horses in Skyrim to this point.
But I have to admit, horseback riding in Skyrim is quite good. It is so much easier to control your horse and makes getting from point A to point B a little faster without resorting to any quick-travel methods, which I mostly avoid anyway.
So now I can traverse the gameworld a bit faster without missing the sights and it is actually enjoyable.
Like many other threads have pointed out already, it would be nice if you could harvest alchemy ingredients and have some ability to combat while mounted, but it is not a deal-breaker for me. In fact, more often than not, having a horse is a great way to just out-run random encounters I'd rather not deal with.
It was a pain at first with my horse constantly getting into fights when I did fight something. But I've learned to just park him away from planned combat situations so that solves that problem.
Anyway, really liking the horses in the game, wish I had simply tried it out sooner.