That svcks in a way. Surely they could put a clue somewhere in-game, like in a book or something? I think without using the Wiki the only way you will come across the Gems is by pure chance. There is no way you could just set out looking for them, with the hundreds of locations...
Even worse, I have it on pretty good authority that at least ONE of the gems is in an area that could be described as a "http://tvtropes.org/pmwiki/pmwiki.php/Main/OneTimeDungeon" (meaning, you can't access that area again after a particular section of the story is complete).
I'm just glad I found it while I was in there. Because if not, It would have been rage on a level equal to when I left Raven Rock without the Energy Weapons bobblehead in Fallout 3. I positively SEETHED over that naked spot on my display stand, and eventually ate about 12 hours of gameplay just to re-load a much older save and go get the S.O.B...