Basically what I'm trying to do is the following. If a character attacks a member of my faction TBJThalmor, or does a particular dialogue option, I want all members of the faction to attack the player on sight. What I've done to further that goal so far:
I have AI settings at Helps Friends and Allies, Attacks Enemies, Foolhardy Confidence, Neutral Mood, and Aggressive.
Their Faction settings include a reference to their own faction as ally, and PlayerFaction as neutral.

During the dialogue in question, I have it as TBJThalmor.SetEnemy(PlayerFaction()), but it doesn't seem to do anything. How do I make it so that when this dialogue is chosen it will permanently set PlayerFaction status as Enemy so that they attack?

Perhaps through Faction.SetActorValue("Aggression",100)
Or you can to create a new faction and in the dialogue to write Actor.SetFactionRank(Faction,0)
In the faction create disposition to player - 100

More problems in this area.
I have a dungeon where there are multiple enemy types, all belonging to my TBJThalmor faction. Unfortunately, whenever I enter the area, they start attacking one another like madmen. Anyone have an idea why?

May be you didn't point disposition to 100. Check tab AI Data, may be your NPC have Aggression status

SetActorValue doesn't seem to work for factions. Whenever I try it I get an error saying "Setactorvalue is not a function or does not exist"
The second idea seems like it would work for individual enemies, but I want the entire faction to hate the player, not to make each individual enemy run a script to make them attack.

What/where is disposition?

In the faction find the form "Members" and add your soldiers. And in the left form add own faction and set disposition to 100

1. Make a faction called MyThalmorEnemies (in the CK)
2. Add the Player to that Faction (in the CK)
3. When you need to, in your quest (script), add MyThalmorEnemies as an ENEMY of TBJThalmor
4. AND on the next line in your script, add TBJThalmor as an ENEMY of MyThalmorEnemies

(you have to set it both ways)

Should work ...

Wait, that sounds like it makes some members of my faction fight the others. That's what I'm trying to avoid, not cause.
EDIT: Nvm I understand now.

I have already done that. Still having problem.

Simply make

Actor.StartCombat(Game.GetPlayer())

Some of them are obviously in opposing factions. You will have to check each of the actors in your scene and see what factions they are members of, then check those factions to see which are set as enemies of the other

Don't think it can be anything else.

BUT DON'T JUST REMOVE THE FACTION SETTINGS.

A better way is to create Duplicates of the "monsters" you want to use and call them a new name (myQuest12IceWraith for example). Then you can make up your own Factions to add these monsters to, without breaking anything else.

I did this and it worked perfectly. BTW rather than create another faction, you can just use playerfaction and reciprocate the setenemy command.

No, they weren't in opposing factions. I think it was just a small glitch, it seems to have resolved itself actually.

Just a tip:

If you create your own factions, then there is less chance that your mod will conflict with some other mod made by someone else.

If you change the Player Faction, any mod that also changes that faction would conflict. the conflict may break your or the other mod.

Where ever possible I create my own stuff (factions, actors, quests, everything), trying to keep edits to the vanilla game to an absolute minimum. If at all possible, the only thing I am editting in Vanilla is the exterior scenes where I must put doors

There will still be conflicts, but you should still try your best to minimize them

Just saying; what you put will work just as well as making up both factions