Nedes arrived in Skyrim to find Falmer and Dwemer. They shared land with each other until the night of tears in which the Falmer attacked and slaughtered most of the nedes. Ysgramor returned with a bunch more and retaliated against the Falmer, who retreated underground. Dwemer enslaved the Falmer. Eventually the dwemer wiped themselves out leaving mutated Falmer and nords as the only occupants of Skyrim. Nords settle in since there is literally no one left in skyrim. Fast forward to the third era. The War of Bend'r-Mahk. Skyrim conquers the reach(Consists of both High Rock and Hammerfell territories). Despite numerous attempts to retake it by both hammerfell and high rock, neither group was ever able to reclaim it. Neither group liked the reachmen living around the area though. Reachmen have a bad rep. Partly due to their mishmash of racial traits, partly due to their practice of hedge magic, and partly due to their sacrificing people to daedra.(They might've been content to let them be with the first two, but that last one probably went over the line)
Plus the reachmen became the forsworn when they made an alliance with the hagravens, who are foul enough.

