I dont see dunmer leaving Windhelm enmasse to go help Tullius. Also, most Nords that are against Ulfric arent actually for Tullius. They just dont want to go to war.
Don't fast travel

I've had a few random encounters with random civilians, including dunmeri, in Estmarch and Rift, who tell me that they are on their way to Solitude to join Legion.
@topic
At first I was neutral, then I had my first Thalmor encounter which pushed me towards Stormcloaks because that early in my game I thought the Thalmor are with the Empire. But then I talked to some people, read some books, and my support for the Empire and pity for the poor Stormcloak soldiers grew stronger and stronger.
The only unsettled issue was my personal one: Empire almost had me killed. I hated Ulfric, but I couldn't forgive Tullius either. That was also solved as soon as I saw them both in person. Ulfric who, unlike any other Jarl, didn't give the tiniest crap about his own hold and even city - young women in his city are being butchered, I solve the crime, Ulfric never even notices any of this - too busy picturing himself on the High Throne; plus he tells me to 'leave my criminal past behind' - hello, lulwut, my only crime was accidentally being near HIM and getting captured and almost executed because of HIM. Tullius and Rikke on the other hand both apologized to me for Helgen.
One more thing that I don't see mentioned (much?) in countless Stormcloak vs Empire threads. If they so love and heed and worship Talos, why don't they fight and rebild and reunite the very same Empire that Talos fought for, built, united and ruled? Instead, with Talos' name on their lips, they help Dominion to ruin completely everything that Talos built.