I know in oblivion armor hurt your spellcasting depending on how high your light/heavy armor skill was. I'm not sure if it's like that for skyrim. Is there a downside to using armor on a mage, and if so, is there any reason to use clothes, or am I best off getting the best armor and enchanting it with awesome stuff?

i think that the only offside of armor when being a caster is that the shield spells dont work as good as with robes, and if you dont level up enchanting itll be really hard for you to get armors with good mage enchantments

Short answer, No.

Long answer, No but one set of Alteration armor bonus spells do not add as much as they would with cloth.

You can play a plate wearing Orc named M-1 Bradley. You will be un-killable. It will just take you half an hour to kill anything with your spells that do not scale.

I use armor and don't have issues. I understand the play in Skyrim is different that in Oblivion as far as armor and mages go.