Just wondering if anything substanstial changes when the Imperials or Stormcloaks gain an enemy territory by trading with each other?

You have to regain that territory eventually if you do the Civil War quests and the guards of the holds change to either Stormcloaks or Imperial guards. Apart from that, not much happens

Practically nothing changes apart from a quest location change during the Civil War quest line.

It svcks because you are forced to change the guards to Imperials or Stormcloaks to further the main quest.

Ok, tyvm all