So I recently put on Master Difficulty and wanted to test my mettle on a dungeon. So I climb up a hill somewhere near the Reach Borders to the back entrance of a fort. I enter it's tower, which is strangely empty, and climb to the top. I summon my Bound Bow and look over the wall. There are only 5 Forsworn in total making up the whole Garrison. 4 Foragers and 1 default Forsworn. 3 of the foragers have bows and the other two have dual-axes. The garrison isn't a problem with my vantage point, so I take them out easily with my bow. Then I summon my Bound Sword and put Firebolt in my left hand and jump down. I get ambushed by a a saber cat which was quite tough, but other than that the fort is empty. No one inside in any of the entrances that I've entered, I look on my map and it doesn't say that I've cleared it yet :/ Any one no why?

Well, it's part of the Civil War questline.

^This.

Many forts are quest locations in the CW storyline, if you never start it they will often be inhabited by bandits instead. If you start the CW they will be taken over by one side or the other and then regarrisoned with the winning side's soldiers at some point. So they are not really clearable, even if you kill all the original bandits the game eventually replaces them with more or with soldiers.