I really enjoyed running around in Oblivion punching out everyone and anything that got in my way.
Yes, you can still use hand to hand in Skyrim but you can't level it, so here's my guide on how to make a bad ass unarmed character!
RACE - You have to go Khajit, they have the permanent boost of Claws which boosts your unarmed damage by 15 points. I know someone might bring up using an Orc, probably because of the Berserk power, but let me remind you, it only lasts for a limited time and you can only use it once a day. Therefore, Khajit is the way to go.
SKILLS & PERKS - As for the skill tree I would suggest focusing on Heavy Armor, Restoration, Smithing, and a little bit of Enchanting along with some Block. Now it might sound like a lot of skills to focus on but let me explain why.
Heavy Armor is a necessity if you are gonna fight with those big ol' fists of yours. It has a perk called Fists of Steel, which makes your unarmed attacks do the damage of your gauntlets armor rating. (only applies to heavy armor gauntlets)
Another perk in the Heavy Armor tree that comes in handy is the first one you can get, Juggernaut which boosts your armor rating with each perk you put in it. So obviously the higher the armor rating, the higher damage your punches do.
Restoration is very helpful for an unarmed character but to an extent. Just level it until you can dual cast Fast Healing, then you're pretty much good on Restoration.
Smithing will bring a smile to your face when you improve your armor using the work bench, just snag the perks lets you forge the heavy armors you might want to use. So you could take your gauntlets from having an armor rating of 20 then bringing it to Legendary which would boost it a crap ton!
Enchanting, you don't need to put much into it, just some so you can enchant your gauntlets with an enchantment that boosts your unarmed damage, I'll get into that a bit later. Also, it might help to put stamina and health enchantments on your other gear.
Block, this ones a keep or throw away, I found that if there's an enemy that's giving me an ass whooping, then putting the shield in my left hand helps when I can stagger him with it then start wailing on him with right hands. Other than that, there are a few handy perks in the Block tree but they aren't a priority.
TIPS - This part is just some useful info that will help you out. Once you escape Helgen and they send you to Whiterun, take the Carriage to Riften and head to the marketplace. There should be a few booth shops and a blacksmith, jump over the railing and into the water. Climb up the steps and head to The Rataway. There's some bandits in there so take them out, one will fight you with his fists so once you beat him, search and grab the Gloves of Pugilist which boosts your unarmed damage by 10 points. Then go to an enchanting table and disenchant so you can throw it on some iron gauntlets or whatever heavy gauntlets you have. Now instead of 10 damage, it will only give you a 5 damage boost but that's because you need to work on enchanting. Just build up enchanting until it's higher than the original Gloves of Pugilist.
Some people have brought up the theory that the perks Dual Flurry and Dual Savagery in the One Handed tree boosts the damage and speed of your unarmed attacks, although I don't thing it does.
Oh and before you ask, yes you can whoop dragon ass with just your fists haha.
So that's my guide, sorry if it was a bit lengthy but I feel like the details count when it comes to unarmed if you want to stand a chance. If you have any questions or need help, just feel free to ask!