The Khajiit have a slightly different hand to hand attack animation, where they leave their hand open to strike (so as to rake with their claws) but I can't find what determines this.

I can't even find where the relevant animation files might be...

If you're talking about switching beast H2H anims onto non-beast races, there was a mod on the Nexus that did this. It was deleted for some reason though.