I seriously doubt you punched things until you were able to get that heavy armor perk AND upgrade your gloves AND enchant them until you were well into your level 20s. And what of the players that do not want to wear heavy armor or use a glove to facilitate their hand to hand combat?
The Khajiit hand-to-hand bonus goes a
long way to allowing you to be Hand-to-hand, serving well into the mid-levels. I get owned by Khajiit bandit thugs that use their claws.
I think H2H ignores armor...