It is not!!! My friends live there!
Oh, as a side note, has anyone really tried to go there not being a vampire? I did and they called me a weak mortal, and said that I should get Serana to cure me at once.
I've broken into the castle a few times as a dawnguard by using the wooden plate whirlwind sprint method.
You can walk around inside the castle, talk to people, take stuff, whatever, no one is hostile.