I think it is a known bug. I've had that since the first days of playing. It happens on average once every 2-3 hours, I think. It is unconvenient to have to restart the game. But at least it's a workaround that works every time. I have the game installed on my SSD, so restarting and loading a savefile takes only 10-20 seconds total.
I must say, since I use TESVAL (the mod that optimizes cpu-usage), I have not had this issue. Might be a coincidence.
Yeah, it's not that big of a deal to restart, but I would like to know what's causing it. I tried the mod you mentioned but it didn't do much (virtually the same fps at the test view in Markarth) for me, so I took it out.
take a look at this thread, hope it helps

http://www.gamesas.com/topic/1323347-how-to-fix-indoorcavedungeon-fps-drop/
Thank you.

I saw this thread, but didn't know if it applied since I get it outside, too. Also, I'm kind of scared to mess with my settings like that.