Yeah, it's possible to lose. At least, I've lost, as an Altmer (twice, on two separate characters) against an Orc (that guy in Cidhna Mine). The two characters in question were a mage (no health to speak of, so not really surprising) and a thief (more stamina than health, for carrying/running purposes), but I think the racial weight differential also made a bit of a difference. I haven't really brawled with them anywhere else, though, so I don't know if wearing armor (light armor, anyway) makes any difference.
This.
My Dunmer Mage got his ass handed to him pretty soundly by the Orc in Cidhna... Only brawl any of my characters have ever lost, but I've got to hand it to him, that Orc put on a clinic that day. lol