I don't think it breaks immersion. It's logical to to believe that over time bandits would go back to previous hidding spots. For a short time that area you cleared would be safer, but to think it will always be safe; I don't agree with. There will always be more bandits. Just like life there are always new crimanols.
I just checked, on my first character, he has existed for 337 days, and has killed 1207 humans. Lets say about 40% of them are bandits (face it, they are the most common human enemy you face, followed by the forsworn) At that rate, they would have to recruit about 1.5 bandits per day, not including the ones that die out from: Exposure, disease, hunger, thirst, in bandit fighting, other adventurers, accidents. They would be constantly finding new bandits on a daily basis, and considering how fast they die out, people would be discouraged from banditry. Also, consider the forsworn, they are native bretons, in order to make a native breton, forsworn have to have kids and wait for them to grow up, a slow process which wouldn't nearly cancle out the rate at which i kill them