I'm sure this is an issue that's been done before, but I just finished the Imperial questline and Hadvar keeps following me around in Whiterun.

-He doesn't follow into any buildings.
-Only appears after Battle of Windhelm.
-If I leave Whiterun, he stops following me and goes to the Sleeping Giant.
-I went with Ralof at Helgan, and interacted with Hadvar during the Battle of Whiterun and the prison break in Winterhold(Rift was in Imp control by "Season Unending", so no wagon despite his reference to it.)

Is there anything I can do to get rid of him? Or if I have to reload a save, how far back should I go?