They are essentially a "uniform", it makes sense given the fact almost every character you make is anti-Thalmor (or will eventually be attacked by the Thalmor) for you to feel apprehensive towards wearing the colors and clothes of the enemy.
Yeah it would be like a Jewish person or an Englishman or Frenchman feeling uneasy about wearing a Nazi uniform. Thalmor robes are the equivalent of military uniforms for magicians.
I generally would only wear them if I'm playing an Altmer or Bosmer, and only wearing Elven Armor for Altmer, Bosmer, or Dunmer.