I was sent to a village in the south to check whats going on there and the Legion was killing all people there (they handed out lottery tickets). I got a quest to follow them and save the bandits. I found them in a small camp, saved the game and then freed the bandits. The Legion NPCs attacked me and I loaded the previous game where the Legion was still neutral but after loading they immediately attacked me! - the Faction was not reset/loaded from the save. I had to exit to the main menu and load the game to fix it.
In the end I found a solution where I could free the bandits and don't lose rep with the Legion.
