As far as I know, modders couldn't create new behavior files (yet), they can just modify the existing behavior files
Animations in Skyrim are more complicated than in previous TES games. They have their own conditions (how to play), and transitions (which animations will play next when the clip's over) and triggers (when to notify other conditions/events while animation is playing)
To be short, defining new animations in defaultmale/female.hkx will just tell the game that this actor (or Race) can play this animation. It doesn't tell the game how & when to play it. You should look at other hkx files (behavior file) for more details
If you want to use existing Race's animation, your race and the race you want to switch to should have the same skeleton rig. Or you will have some problem. If you satisfy the above condition, the easiest way to play new Race's animation is to make the new race use 'Morph Race' data in the CK
EDIT: Another easy way to add new animations without messing with the defaultfemale/male.hkx is to use "fore custom animation" mod
To clarify, modders cannot create new behaviors solely from within the creation kit. It is believed that Bethesda wanted to avoid the explosion of six-based mods like what happened with New Vegas. However, it is possible to create new behaviors now, and you can do so using the Havok behavior tool, which is conveniently being offered for free at the moment. A guy by the name of
Theru posted an introduction of sorts of how to create new behaviors over at the nexus, go http://forums.nexusmods.com/index.php?/topic/624251-progress-in-behavior-filesalmost-done/ i you want to see the original post.