At least for my character. I'm not using a full set of heavy or light armor, I don't wear the helmets. Should I even bother with the perks, or can my smithing max out the armor any way?

Don't need perks. http://www.uesp.net/wiki/Skyrim:Armor_Rating#Armor_Rating

With enough fortify smithing armor perks aren't necessary for boosting your defense.

However if your say a dual wielder in light armor but won't want to exploit the crafting system for a fortify smithing 200%+ you'll probably need some armor perks to get your defence up. If your a heavy armor shield user. You probably won't need any heavy armor perks.