I believe the game is checking to see if the player has 5000 gold and then is deducting 8000; possibly because the script may have player money check remnants of the Whiterun home buying script which did cost 5000.
Simple to reproduce, just have less than 8000 gold but more than 5000 gold when you go to buy the Riften home, and the game will allow the purchase anyway.
