I think it's because seeing your enemy walking around you doesn't really add to immersion. They probably want the only encounters you have with opposite factions to be on the battlefield.
I can respect that, TBH.
Well, for one, it already doesn't make sense that people from opposing factions can walk into an enemy faction with no consequences, and second, it would further aggravate the open-world PvPers who would say something along the lines of "No fair! We can see them, but we can't attack them!"
You could just have some channels that are cross faction, and some that are not.
So if you don't want to see people from other factions, that's ok. If you do that's ok too.
Everyone is happy. Well except the your fun is bad wrong people.