I was in a cave on the eastern side of the island and came across a cave. Thr room before you clear it had many rieklimgs worshiping a horse and cart. It made me chuckle a bit but then I relized I have no idea what their culture is like. A loading screen says that they don't have much of one but obviously there's something to it. I just wanted to know more about them if anyone knows

They attach themselves to the "cultures" that they find. I assume this is why so many carry statues of Dibella and can be found worshipping random idols.

The place you speak of is located in Frossel. It's some sort of elk or deer made from bones, junk and snow and there ooks like some sort of writing painted on the cart in a green kindof paint. The whole thing is pretty detailed, and even has a little pile of charcoal poo behind the deer.
For some reason the scene made me think of the Rieklings worshiping Santa.

Isn't it just a horse from a horse and cart that fell through the huge gap in the ceiling of the cave?

Take a look at the appropriately placed charcoal around the horse. Definately a good chuckly moment. Nicely done Bethesda, nicely done indeed.