It seems to only do it after I go to jail at a hold with a $10000 bounty
I don't know if there is a relation between it...I believe I have patch 1.6 (mounted combat)
But to be more precise here's what happens..
Go to town with huge bounty.
Accept the jail time.
Sleep in jail.
Then its the super long load time.
But I once waited it out and the game was super laggy and didn't want to move, so I waited and waited till I could get to the map then I fast traveled away (stilltook a while) and the went back amd it was back to normal. But then I tried again at another hold, and it just took so long I gave up...