I had the same thing happen. Never found a fix, deleted the save and it went away after patch 1.6.
Came back with Dawnguard when I couldn't load my main guy. No fix either, had to delete and roll back.
Use MANY save slots so you CAN roll back.
Losing 20 hours svcks, losing your character svcks worse.
.... Ok here is what I did and didnt do lol... well first turned save on travel off... I also noticed I was trying to travel through front door ...

Using the door at the back near the top of the map let me right in.... I was trying the front and would get the freeze, so it was either neither or both things that helped.... Thanks to everyone that helped, I had a ton of stuff in there inculding the jester outfit
