Why would they be there at all if you weren't supposed to find them? Plus the items are not marked as stolen when you take them, right?

Because the way the merchants work in Skyrim, Oblivion and Fallout is that the items they sell are stored in a container which is linked to the merchants. The chests in most cases are hidden just outside of the accesible bounds of the map, but kept reasonably near the merchant for the sake of making development nice and easy; need to edit the merchant's stock? - just click the camera under where he works.

This holds true of the khajiit in Skyrim; but the bounding box of the chest is slightly larger than the actual physical chest. The chest is hidden just below the world mesh, but the bounding box pokes out the ground a little and it's that bounding box you're able to grab in this example. This is definatly NOT an intended feature

Actually, it's totally a glitch. The chests were obviously meant to be under the ground where the player cannot access them (this is how merchants store their inventory in Skyrim, Oblivion, and FO3), but Bethesda didn't place them far enough into the ground, creating this glitch.

That glitch was fun! Your bow would shoot as many duplicated items as arrows you had equipped, so you could make stuff go flying everywhere.

More like, containers not placed far enough under the landscape. They don't have to be there though, or maybe there is a reason they have to be near THESE merchants.