Is there a way to fast travel there? Everytime I leave and need to go back, I have to walk all the way there from Riften... If this by design or a glitch?

Er you should spawn infront of the castle and from THERE you walk a bit

But Fast Travelling to Dawnguard makes Skyrim not as fun, in my opinion it is much more fun to Walk around Skyrim instead of fast travelling when there is new things to be killed by.