Well, thats certainly an immersion killer.
Really? There are like 50 horses in the entire country of skyrim, of course most bandits aren't riding one. The only people you see using them are nobles, guys pulling carts for cash (which are expensive), folks who own stables, and imperial soldiers or stormcloaks who probably stole them from the imperials.
In my opinion it would be more of an immersion killer if there were multiple bandits running around on horseback, when its pretty clear only the richest of the rich people in Skyrim can afford horses.