Imperials are out since they have heavy military presence in Skyrim
High Elves are out since they have considerable presence because as Thalmor and as mages(particularly as court magicians)
Dark Elves are out since dragonborn takes place in Solstheim which is inhabited by Nords and Dark Elves
Orcs are out because many strongholds exist that are only Orc-populated
Khajjits are out since many of them exist in camps outside cities
This leaves Bretons, Redguards, Wood Elves and Argonians. I believe Argonians are fewer but i might be wrong