It does take away from the pure mage aspect, because you are required to place perks in the corresponding weapon tree to make them work effectively.
.. but at the same time, it doesn't because your weapons are magical, and not mundane.
I have a character that only uses Thu'um, lightning magic, Storm Atronachs, and bound weapons. Half of his perks are in combat oriented trees, but I still consider him a sorcerer of sorts, because he doesn't carry any mundane weapons, besides his fists.