Not new actually, just have a terrible case of forgetfulness.
At any rate I'd appreciate a taming route which allows to do such things without specifically using magic.

Like that movie, "Beast Master"?

That horse was owned by soldiers not bandits, so you are still stealing it, but you should be able to take it if there isn't anyone around IMO.

I wish there was more animals you could have as a follower though. Maybe a quest that allows you to raise a bear cub, that grows into a bear you can ride.

No, not really. Dar can communicate telepathically with animals in that film.
So unless you can conjure up a known TES alchemical ingredient which can induce telepathy with surrounding beings,
it'd still count as magic I believe. But taming animals without magic spells should still be possible, just got to utilize it's gameplay with Speech dialogue instead.

I am porting my Master Summon mod to Skyrim in the form of a Hunter class. You will be able to permanently tame and re-summon any animal in the game (though you will have to reach a certain skill/level for higher level creatures.)

Same as my summon mod (see my sig), you will also be able to summon multiples of lower level animals (think personal wolf pack), as well as have total control, being able to tell them to attack specific NPC's. All the same functionality of my summon mod, including advanced follow AI, spell to port them to you, hotkeyed favorites, tell them to wait, patrol, etc.

Since all the code is written it shouldn't take too long to port it once the CS is released.

Ideally, I would be able to add an entirely new perk tree to handle certain features...