Basically.. I am roleplaying what I like to call a Witch Vanquisher. Heavy armor, shield and mace and uses NO magic(I am playing as a redguard and in their lore it says they dislike magic, so I rolled with it.) Anyway, I use alchemy to keep myself in good standing. Make drain magicka potions for pesky mages! Make resist magic potions also and I took the block perk to resist magic even further.

Now my only problem is.. I feel like maces are not as good as swords or axes.. they swing so very slow and it seems they don't get much more out of it? Also even with all this resist magic stuff.. mages still are a huge annoyance to me. They all have a dang frost spell for some reason... I am obviously a more warrior like character and don't really have a gap closer until later. Anyway, is this a bad character design? I really like my character so far(I am level 20.) Dwarven everything.

Anyway.. Is basically throwing out magic a good idea? or will the game get difficult for me if I don't use it? SO far the only thing that really gives me problems are swarms of enemies. It seems my mace does so little damage.

You can decrease the difficulty level of the game. My equipment is, for the most part, unenchanted and outside of the shouts, I don't use magic much. That said, magic does play an important role for me, it's often my safety net. Also, if you're avoiding shouts, that can be a game changer. Without shouts I would die far more frequently. They save me, especially when I'm getting swarmed. Breathing fire on a group of enemies changes their outlook on life pretty quickly, and their health bar VERY quickly.

oh I still use shouts. Unrelenting force is my most useful one it seems.. although I probably should start using whirlwind sprint to chase those mages