I've seen bandits, or their remnants, attack smaller settlements like Riverwood, but it'd be pretty foolish to even try attacking a walled city without large siege weapons. Bandits are pretty much like any predator: seek the most gain from the least risk. The more guards, the more risk. And even if they succeded in attacking, say, Riverwood, any survivors mean the Jarl is likely to put one heck of a bounty on their heads -- one sizable enough that even the dragonborn might be interested.
