The game is just full of bandits and you will rarely find a friendly cave or fort. Bandit killing is a pass time in this game.

Bah, it's sad to know, sometimes I come upon a fort after a long, tiring journey just to be attacked by strong bandits..
That's why I wish there are some friendly forts that allow travelers to rest/eat (I know there are villages, but forts can be a nice change.) -- without doing certain quests.
When you do the civil war quests the forts you take will be friendly.
I've just started civil war quests, I'm looking forward to find those friendly forts now.

There is a cave full of Foresworn that are friendly if you do a certain quest a certain way.
It is kind of weird to talk to them for once.
I have done the quest with friendly forsworns, (but not in caves yet) and yes I agree, it's weird talking to them instead of trying to chop their heads off.
