I've seen them in Whiterun, Markarth, and Solitude. They're like house numbers I think, and I'm pretty sure that with in a town they're all unique. Any else seen them and know what they're for?

Yes, they are called Shadow Marks. More information available at your nearest Thieves Guild hall

Ah, that explains it, I haven't done the thieves guild yet.

Search through various merchants or keep an eye open for a book called "Shadowmarks."

A full explanation is contained within.