The only time an attack has come close to a possible issue is when I was lvl 8. A pair of vamps attacked Riften, and cleared out the market circle along with a few guards, and half of those that were inside the Bee and Barb. During the day.

...it was awesome to watch.
I would have left it like that since I never use Riften (aside from visiting the thieves guild from time to time), but for some reason I decided to step in and take a few swipes. Only succeeded in pissing one off, and then promptly joined the dead market circle. When I reloaded I figured I wasn't ready, so I just avoided going outside (waited 13 hours). I was all clear, they didn't spawn, no one died.
Other than that, my fights have been painless because I've been at higher levels. If the guards and NPCs don't take care of them, it only takes two or three swipes and they're down.
