Yeah, this is what Josh Sawyer was talking about last year when he said that the way Bethesda's engine handled saved games was terribly inefficient in terms of memory. Basically every time you enter a location an index of the default for that location is created within your save file and everything you do in that location is saved as a bit difference from the default. Essentially this means that whenever a location is visited the game has to load that location twice, first the default and then apply the bit differences to make it the way you left it.
Factor in Bethesda's other bad memory design such as dragons and active quest characters being treated as near distance actors regardless of their global position and their lack of optimisation to our hardware and there's your reason for the frame rate.