From what we saw, there was the Skaal (a single village) and a handful of outcasts. One of these was a budding necromancer, and the other three knew how to use Stahlrim as a material for weapons and armor.
Those latter three were kicked out because using Stahlrim as anything other than caskets is apparantly a blasphemy to the Skaal.