For entirely no explanation, there are giants protecting Kolskeggr Mine. Not the mine itself, but the surrounding area. And yes, I said "there are", as in there are 2 giants doing it.

And they're not just "happened to pass by", they're actively protecting the area from Forsworns. One protects the path to the smelter, while the other one patrols the area around the bridge. They're not going anywhere either, just walking back and forth around their patrol area. I've visited Kolskeggr twice, and they're still around.

The first time I visited, the forsworns around the area were already dead. In fact, I witnessed one of them being mauled by the giant near the bridge.

Needless to say, it makes the conversation between the miners about "the forsworn won't dare to attack again" so much better.