I do not care for the racism and segregation of Windhelm, though.
The Dunmer live in the Grey Quarter because that section of the city was given to them, free and clear, over 150 years ago as a place to create a self-contained and self-regulating Dunmer enclave where they could settle in Skyrim without swearing fealty or becoming beholden to Skyrim's governing authorities (in this case, the Jarls of Windhelm). They are not forced to live there - they are free to leave at any time and make a life elsewhere. Have any of them been able to buy property somewhere else inside the city walls? No, but that's not a "right" that someone is denying them, it's a privilege they haven't earned. That privilege is granted by the Jarl of Windhelm as a reward for services rendered to him personally as well as to the other residents of the hold. You don't get that privilege unless you've proven your loyalty to the Jarl and gone "above and beyond" when it comes to the well-being of his people.
Where Jarl Ulfric is concerned, that includes being willing to prove one's loyalty by fighting for Skyrim's independence and, if necessary, to defend his land and that of his allied Jarls against Imperial incursions. This is something the Dunmer are simply not willing to do because they think it's not their fight and not their problem. One of them will also tell you that they don't care about the murders being committed in the city because the victims were Nords, not Dunmer. They seem to want to keep themselves a "separate people" within Skyrim and Windhelm, and that was part of the agreement under which they were granted a settlement there - but they don't get to do that and still enjoy the same privileges as people who have actually earned them.
As for the Argonians, they are kept out of the city for their own safety, as even Brunwulf Free-Winter will tell you if you take the city for the Legion and he becomes the new Jarl. And BTW, all of his talk about wanting to help the Dunmer yada yada yada appears to be just that: talk. When Ulfric is Jarl you can hear him promising the Dunmer that he'll go to the Palace and speak on their behalf, but all he does when he goes there is stuff his face with free food and drink at Ulfric's expense and then leave without saying anything. If he becomes Jarl he talks a decent game but nothing actually changes.
And regarding the Khajiit, they are treated no differently in Windhelm than in any other hold/city, so Ulfric is no better or worse than his fellow Jarls in that respect.