So Does anyone know how to buy clay without the steward

You should be able to mine it somewhere around the house. I found mine next to the stables, or just to the right of the building station.

The clay deposits are usually right next to the house or where the house will be. They are minable with a pickaxe, and they supply unlimited amounts of clay, 4 "ingots" at a time (you don't have to process, refine, or smelt the clay in any way; when you mine it, the clay comes out in these neat little bricks!). It's quicker to just whack at them without going into the mining animation, as usual. The same is true for quarried stone; there ought to be a stone or rock somewhere near the house that looks whiter & cracked that is an unlimited minable rock quarry.