They are racial alliances after all, and anything else would be really difficult to justify.
No. No they are not. Racial that, is. They are regional alliances. The Ebonheart Pact was created to defend the borders of Skyrim, Morrowind and Dark Marsh from the invaders that constantly push into their territories. Doesn't matter what race you are in that aspect; you could just as well hate a particular faction for bothering the city you inhabit. Whether it is part of your race or not. The Daggerfall Covenant is based on economy, which again is based on location, not race. Finally, the Aldmeri Dominion believe the world's faults to be caused by disorganization and that particular races are more chaotic than others. Men being the worst of them. So, while the Aldmeri Dominion is racest, it strives for racial hierarchy rather than exclusion of other races.
Although I do love exiles as a type of 'faction', where it's free for all.
On the note of neutrality; I think that most peasants and townsfolk would be considered neutral, while their protectors are considered to be of a particular faction. The townsfolk defend themselves when they need to and can accept new leadership when such establishes itself. It's primarily the military that should be forced to pick sides, while there might be a few fanatics among the villagers that wave their flags at any occasion.