I've noticed that if a guard starts to talk to you when you enter Breezehome he will be transported to the interior as well so that he can finish telling you about his knee. They they leave.
I'm LOLing!
But it is true, if someone starts talking to you, they tend to follow you inside of wherever you're going, I had this happen a ton of times using
that ladder exit in the Thieves Guild.
Apparently the NPC's think their stories are so important they have to follow me, even though they've told me the same story over 200 times now.