The one server to rule them all idea is horrible as it will be really laggy.
As an Australian i get horrible lag but can play games like WoW and GW2 with 100-200ms ping.
But with one server wouldnt the ping for those outside of america be like 300-500 ping?
1. There will most likely be different megaservers for each sort of region of the world.
2. Don't think of it as "One huge single server" instead try to think of it more as "A bunch of servers within one single server" like the basic idea is that the game still has servers, it;s just transitioning between different ones is much easier now and some have certain preferences/restriction settings to create certain player communities and so on.
ZOS wouldn't be doing this is they weren't confident that it wouldn't lag don't worry.
Also as a sidenote, the overflow system from GW2 is KIND OF like what is being done with ESO. Same general idea.