Actually, Molag Bal's symbol is skeletal-like and monstrous-looking.
His symbol could be many things. Every Daedra / Divine has several emblems or icons that represent them. Take Diabella for example. You have that naked statue, the type of flower depicted in that statue (not sure of its name), the shrine, and the amulet (which does take on the aspects of her shrine).
What I've found is this.
http://images3.wikia.nocookie.net/__cb20120612162149/elderscrolls/images/e/e2/Dawnguardvampiremaster.png is a shot of Harkon. His broach bares the face of the bloody shrine in the cathedral.
The bloody shrine is a take on the vanilla version of the shrine in Markarth as seen http://images4.wikia.nocookie.net/__cb20120513042927/elderscrolls/images/c/cc/Molag_bal_mace_altar_rusty.png.
The mace also bears the face of the shrine as seen http://images.wikia.com/elderscrolls/ru/images/e/ec/Mace_of_Molag_Bal.png. But also notice the spiked edges of the mace and shrine.
I think those spikes carry over nicely to the broaches worn by Serana http://skyrim.nexusmods.com/Images/1207799-1345615689.jpg, the Vampire Royal Armor http://www.skyrimdl.com/wp-content/uploads/2012/08/Dawnguard-Vampire-Armors-Mod-at-Skyrim-DL.jpg, and the Vampire Armor http://static.skyrim.nexusmods.com/mods/images/21637-3-1344048558.jpg. Obviously these broaches are a take on the chaos symbol (

) but it still could be (and imo is) another symbol of Molag Bal.
Interestingly enough, that star symbol
could also be a perversion of the Dawnguard's sun / light symbol, though I couldn't find any 'good' link to an image...but we'll go with https://images.nonexiste.net/popular/wp-content/uploads/2012/08/The-Dawnguard-heavy-armor-set-looks-amazing.jpeg for now.
[Edit] Another thing just occurred to me. Take another look at the Vampire Royal armor worn by Serana (and you, same thing). Notice that her collar has a similar demon-skull look as her fathers broach, and the shrines. It could also be that the Royal Armor is an 'artistic transition' deliberately developed by Beth to indicate a transition from one emblem to the other, yet maintain the continuity of their symbolism. Perhaps a little deep, but I find it interesting nonetheless.
