Okay, I've been playing a Vampire Char lately and figured out Dawnguard will randomly pop up and attack you... And it yes, they are hard to kill since I am still a low level..
What really annoys me is every time when I travel to Castle Volkihar, or just go outside (When I was inside b4), there is always another group of them right there on the bridge waiting for me..
I don't mind the Dawnguard too muc. It makes sense, they are vampire hunters.
You know what's much worse? Random vampire attacks in the towns... because so many NPCs are getting killed! I have lost two Merchants from Windhelm and Markath already. At least the Dawnguard are not killing people in the streets!
I have been attacked on my VL char by the Dawnguard and by vampire on my Dawnguard char, but I do not think it is annying. However, the Dawnguard Scouts are places just a little to close to the FT point of Castle Volkihar IMO, unlike the Vampire Scouts that are placed a decent distance from the Fort Dawnguard FT point.
I'm one of those people who don't get Vampire Attacks on my Main Guy.. But the Dawnguard are annoying me cause of them constantly being at the castle... It isn't because of them in-general.