Armor does slow you down, but both Light and Heavy armor have perks that cause the armor to no longer do that.
I think you can sprint longer and sneak a little better in clothing, but I'm not certain. And I'm not sure if those aforementioned weightless perks negate the sprinting penalty, and that's assuming it really is there, since I'm going by the UESP wiki which is not entirely accurate right now.
If you're not wearing any armor, you can get an Alteration perk which massively improves the power of the protection spells.
If you use only a main-hand weapon and nothing in the off-hand, you can block and bash with the weapon just like a two-hander. Problem is, the game has a bug that makes it impossible to empty a spell from a hand slot without equipping then un-equipping a weapon to that slot, and I would recommend using some Alteration and Illusion in the off-hand every once in a while.

If you really want a swashbuckling type character though, I think you should use either two swords, or a sword and a dagger. It would be even better if you occasionally swapped the off-hand dagger with an Alteration/Illusion spell like I mentioned, especially to activate a protection spell.
The dagger in the off-hand would not only fit the Swashbuckler thing best I think, but I believe it would make your sneak attacks stronger. And one of the benefits of clothing I think would be the slightly better sneaking.

Well, it's for sure that heavy armor makes more noise then light armor (check the notes at http://www.uesp.net/wiki/Skyrim:Heavy_Armor)
I can't find anything being mentioned about no armor at all though, though I suspect it's a weight related thing, as the no weight perks of light and heavy armor also take away the sneak penalties. Could be the lighter the clothes you are wearing, the less penalty to sneak, not 100% sure on that.

There is no benefit to not having a shield, unfortunately.

Hmm, small stamina and speed would be good, not sure if they were implemented though. The only noticable benefit I've seen is much less encumberance of course, even with the correct light/heavy armor perks armor still weighs your inventory down, the encumberence removal only applies to ability to be quiet when moving, it still weighs full amount. As for no blocking, well you can punch and use torches and dual wield, 2 cutlass' for a swashbuckler still looks pretty awesome

Natural armor bonus? No, no, no, no, no. No, no. (No.)

The point of going unarmored, is not being hit. You dodge, you dance around your opponent, you sneak on him... but if you get cornered by a group of bandits, you're basically dead. Or you run. Having an "armor bonus", correct me if I am wrong, would be a damage reduction if you are hit. But being a talented unarmored fighter doesn't mean you magically grow a tougher skin. The perk should therefore be, say, an increased chance to dodge. But if you get hit, you're still mincemeat.

I say this because, if you give an increased armor bonus to unarmored and a reduction in weight to heavy armor... well, then it basically plays out the same as you get the high-level perks, doesn't it? Whereas if you give an increased chance to dodge... you'll still get slaughtered if you get hit. So you have to change your playstyle: if you wear heavy armor you attack, assault, tank... and if you go unarmored, you pick your fights in single combat, run from groups, or learn to dance around them trying to avoid their blows.

Enchanting. When I RP, I stick to what I want. If this means a http://i208.photobucket.com/albums/bb319/joniscrazy/qwqwww.jpg wearing clothes over armor, then that's what I do. So in essence, enchant to stick to your aesthetic build.