Skyrim isn't really designed to play as a musketeer (no armor, light sword, etc.). Why no armor when most historical swordsmen did wear armor and if you're not wearing a shield, why not a two handed sword?
Do yourself a favor and wear at least light armor and if you don't want to block, get yourself something two handed.
I'll say too there are armors (light armors) that look pretty much like clothes (rather than armor) like the dark brotherhood armor, the forsworn armor, etc.
The problem with your strategy is that you want to fight close (where you will take blows) but have no armor to protect yourself. If you were using a bow or magic (or stealth) this might make sense. But if you want to be a warrior that fights close up you have to be able to take a blow or two. Otherwise the first bandit chief will one-shot you.
I am considering to make use of Light Armour. I fear that the Heavy counterpart is quite anti-climatic to a Swordsman, they are aligned with Warriors if you understand my concept differentiation. I will use a sword to block, lacking a shield, because I find claymores, two handed swords quite non-swordsman. They just look plain, simple, not requiring skill or strategy.