To me it somewhat looks like the head of an insect, with mandibles and antennas, like chaurus hunters (i saw insect wing bridge in Apocrypha)

It represents Hermaeus-Mora, the Daedric Prince of fate and forbidden knowledge.

That's what he looks like?

He looks like agiant green mass of tentacles. The mask resembles an octopus... kinda

Swarm of tentacles and unimaginable horrors from a dimension alien to our perception and sense of morality? Pretty much.

I say it just has a free-flowing, serpentine design. It doesn't have to be related to Herma-Mora. Maybe it's just highlighting the Dragon-Human duality of the Dragonborn that the other Dragon priest masks, and themselves, didn't have...

The Dragonborn mask?
Every time I see it, I see female genitalia.