The Falmer seem to be little more than animals at this point in their regression.
The Forsworn are actually people, with a culture, history and so forth.
This is precisely why the Forsworn should be detested more. You don't "blame" a Sabre Cat for trying to kill you. The Falmer have practically devolved into an animal state where conscience is non-existent. The Forsworn are not there yet. Yet they are hardly different in thier actions, still engage in self-mutilation, human and animal sacrifice, and they keep evil witch haagravens in their employ. They attack everyone on sight with the intent to kill, be he innocent or not. They consipire to kill those who do not even stumble upon their twisted camps (killing innocent people in Markarth) which shows that they are fully conscious of their doings.
It's hard to hate them more than the Thalmor, however, but I still do. The thing with the Thalmor... they are deluded with pride and vanity, but it's their honest belief that they are right that drives them to persecute so many others. For example, they believe that the worship of Talos is heresy and sacrilege all in one because they believe that it's impossible for a man to have become a god, and certainly not equivalent to their gods. They may be wrong, but they are convicted they are right and instead of attempting to use persuasion and philosophical/theological debate to convince Talos worshippers that they are, they choose to use brute force and legislation (probably because their position is unconvincing). It's a mistake, for sure, but they aren't as disgustingly evil as the Forsworn. Thier arrogance doesn't help them here though.
As for Bandits/Outlaws. I definitely don't hate them, but they happen to be sitting on loot that I want. I look at them as adventurers like myself except they've formed a band and have laid stake to a claim over property that doesn't belong to them. They are delusional, indeed.