Actually, even the Imperial Holds keep the Khajiit out. Not sure why, but that is the case. As for the Argonians I am not sure what to think about it. There is one source in the game that gives any view of it at all, and even if the Empire wins the Civil War they stay out of the city. I would like to se more sources on this before I proclaim Ulfric a racist.
It's something to do with moon sugar and skooma I believe. An attempt to halt it's trade in Skyrim, as well as the general perceptions that Khajiit are all thieves.
And like I said the Argonians are kept out of Windhelm for their own protection. It's like segragation, it's not desirable but it's necessary at the current moment or Argonian blood would be washing the streets of Windhelm. That's not Ulfrics fault, that's just due to racial tension in this particular part of Skyrim. It existed long before he came to power, the Dunmer were even driven out of Winterhold by the native Nords a while back.