No, he came to skyrim and murdered the elves so he could settle on their land. Then the elves retaliated. You are also forgetting that he dwemer are elves as well.
Before Ysgramor, the Falmer & Nords coexisted. Yes, the Falmer were there first, but they managed to live side by side. Then the elves, feeling threatened by the Nords numbers, began exterminating them. It was only after the 'Night of Tears' that Ysgramor fled back to Atmoro, then came back to drive the Falmer out. The Falmer drew first blood.
