I'm working on a new race mod, which I would like to have start the game hated by pretty much everyone in the game. Anyone the player encounters should flee or attack this race on sight... except it's own race of course.

Any help on how I can accomplish this?

Easiest way is factions. But it may not work for you?

Stick each NPC of your new race in a New Faction that is "hated" by all other factions, depending on how aggressive another NPC is (i.e. one not of your new race) they will attack your new race on sight.

(if you also need them to be members of other factions, that may stop you doing this, or at least stop it working, though)

Whether an NPC of your new race flees - or fights - is down to the combat-settings of that NPC.