Everytime I fast travel I see Durnehivrr fly over me and going in circles has this happen to anyone I found it really cool durning the battle for Solitude

Not a gltich. When you summon him, he will continue to follow you until he is defeated.

Really? Because if i don't wait or fasttravel then he'll glow purple and disappear just before i'm able to shout again, note that i'm using the Amulet of Talos.

Oh, and i also summoned him inside Fort Dawnguard once.

I've experienced this too. Worst case was when i fast traveled to fort dawnguard and ended up merged with durnehivvr so i couldn't move anywhere. -.-
Tell me that's not a bug.