Not completely sure but if you wear full set of daedric armor, the vigilants see you as daedra and attack. I'd love to have some kind of voice effect on my character when I wear the helmet. What if daedric armor was a curse that you couldn't unequip without magickal means or ritual but it still would grant it's wearer immense power.
I would imagine most people would not equip it without having gained the means or the power to unequip it at leisure, in which case it just becomes like any other armour.
If this a question about how the Vigilants react, or should react, well... conjuring Daedra or wearing Daedric armor doesn't necessarily make you a Daedra worshiper... does it? Carrying Daedric Artifacts on the other hand, should provoke at least some sort of reaction.
If this is a wish for Daedric armor to become godly equipment, at a price, I believe this is what the modding community is for. But, on the other hand, there are ways to become a demigod in your game so...