Actually, I think it's more likely to be fear. Humans hunted them to near extinction, elves destroyed their sight, body, minds and culture. They're savage, fearful creatures. Pitiful, but still dangerous.
Very true. Are Falmer misunderstood? Nope. Are they the victims of their ancestors stupidity? Absolutely.
Falmer used to be Snow Elves, the original native Aldmeri which occupied Skyrim before the first humans immigrated from Atmora to Skyrim in Tamriel
The Snow Elves initially accepted the first Atmoran immigrants into Skyrim. For several centuries Man and Mer coexisted peacefully.
Until the Aldmeri began to notice how poorly the human race was being assimilated in Skyrim (and throughout Tamriel).
They became alarmed at the fact humans were reproducing at significantly faster rates relative to the Mer. The human settlers also tended to be xenophobic, (aka prejudicial about accepting cultures that were not their own).
This sort prejudicial bias was especially true of those human descendants who chose to remain in Skyrim. These Nord ancestors were the most culturally egocentric of humans in Tamriel. They were the most resistant to any cultural assimilation or integration attempts by the Elves--a prejudicial view which persists in Skyrim to this day. This xenophobic attitude had a lot to do with Skyrim's close proximity to the Atmora mainland (which is to the immediate north of Skyrim).
The Aldmeri across Tamriel grew increasingly alarmed by human irreverence for Mer culture or customs. Particulary about their exploding population growth. They feared the power balance across Tamriel was shifting in favor of the humans. In Skyrim, the King of the Snow Elves --aka the Snow Prince--became the most alarmed of all the Elven leaders.
One night, the Snow King gave into his paranoia and attempted complete genocide of mankind in Skyrim. His psychotic murder spree nearly succeeded in annihilating the human race in Skyrim. All save 3 humans: Ysgramor and his two sons.
Swearing revenge, Ysgramor escaped and fled to Atmora. He gathered his legendary 500 companions, returned to Tamriel, and started Armageddon in Skyrim. The Snow Elven King was killed in the Battle of Moesring Mountain in Solstheim (near Morrowind border) http://www.uesp.net/wiki/Bloodmoon:Moesring_Mountains His defeat scattered the Snow Elves all across Skyrim and as far as Solstheim. Ysgramor and his companions proceeded to purge Skyrim of all Snow Elves. Most of the Elves who survived this massacre fled underground and sought help from their cousins the Dwemer. Big mistake.
The Snow Elves were doubly screwed by their Dwemer kinsmen. First they were tricked into eating poisonous mushrooms and becoming blind. This turned out to be a hereditary condition. Then their Dwemer kinsmen decided add insult to injury by enslaving them. After centuries of abuse, the Snow Elves lost their Elven identity and degenerated into the souless creatures known as Falmer.
To their credit, the Falmer eventually rebelled in a major subterranean war. They overthrew their Dwemer masters and fled deep into the Blackreach. The rest is history.
The wretched degenerate creatures you get the misfortune of meeting centuries later in Blackreach (and in Morrowind) is what's left of this once proud and noble Aldmeri race. Which is really too bad. From the TG quest line, their King was actually a rather handsome looking chap for an Aldmeri.
Bad Karma payback can be a real b*.itch. Especially since the victors get to write history.
Nord version of why the Falmer svck http://www.uesp.net/wiki/Lore:Fall_of_the_Snow_Prince
Also if you played Morrowind, you get a quest in Bloodmoon that gives you an Easter Egg of what was to come in Skyrim here http://www.uesp.net/wiki/Bloodmoon:In_Search_of_the_Falmer
The quest leads you to the Snow Prince's final resting place in the Moesring Mountains, where he met his most sorry and anticlimactic end.