I did the quest where the guy in the Bee and Barb in Riften send me to steal a horse for him. First, some spoilers:

1) If you kill one of the guards outside he is likely to have a house key on him.
2) Four guards rushed me as soon as I entered and started snooping around. Killed them all, picked up a bunch of nice steel plate armor. I was overburdened, and tried to give some to my companion, but she criticized me for even being in there, and refused to take stuff from me.
3) You need to steal the horse's ownership papers. They are downstairs, laying out in the open.
4) When you steal the horse, note that your companion on foot can not run that fast.

Anyway, I took Frost the stolen horse to the nearby location where my employer was waiting. In my dialog choices, I was able to choose to persuede him to simply give me the horse. I suppose anyone could simply kill him to keep the horse. You have the ownership papers, so he is not "stolen" anymore, he is yours.

Now my questions:

How do you "park" a horse? I don't want to lose him. I see no obvious way to leave him safely at the stable. Will he stay put, or wander off if I go dungeon diving? In short, what is the deal with horse ownership?

I have to say that riding a horse in Skyrim is much nicer than in Oblivion. You can steer him the way you steer yourself.