No, folks like Tolfdir just never followed the rest of their kind to Cyrodill.
What do you mean by that?
Anyone notice how racist they are in Windhelm towards dark elves? I'm Breton, and I actually fist faught a guy because he was talking so much smack about dark elves.
Maybe they find it hard to look fondly on the Dunmer because they've been going on slave raids for centuries?

Come to think of it, for how racist the Nords apparently are, have they ever been explicitly mentioned as participating in slavery? I don't believe the lore ever mentioned the Nords widely owning slaves, but http://uesp.net/wiki/Lore:Slavery and the Dunmer still practice it. Even if the Nords did this too, it's pretty clear from this that they're not unique in their racism, and at least they're not slave-owners.