You know that there are about 2000 bandits, vampires, evil necromancers, stormcloaks, other nasty people and random travellers for every house in Skyrim and it never made sense how only 2% have an explained backstory/place they grew up and lived well i thought about how it makes sense... Maybe we all underestimated the size of the original Winterhold. The college has an underground place, perhaps the city had layers, just a thought probably makes up for another 10% at best, Could also help with some people's role plays.

Note: I didn't mention Forsworn or legionares becuase forsworn are from the Reach and legionares could have been drafted in from Cyrodill