There is a house you can get for free right away in Markarth from the "House of Horrors" quest.
Nothing in the house respawns, and there are a bunch of containers.
That is usually the first house I use until I buy a more suitable one.
Talk to the Vigilant of Stendarr who asks you if you have seen anyone entering or leaving an abandoned house.
Outside the Warmaidens, there are two barrels where you can safely store stuff until you get a house. Keep in mind that if you are going to oufit Breezehome, you will need about another 2000 in gold on top of the 5000 price tag.
I actually used the barrels there before, but if you don't go back to Whiterun after a month or so, the barrels reset and you lose your stuff. I lost all my crafting mats after a stint in the Dragonreach prison
