I actually found the first one on accident, in the Dwemer ruins where all of the interactive environment puzzles are (not the ruins where you take the trials---sorry, I cant remember how to spell the titles of these locations at the moment, don't have the game running...), and this was before I even knew what they were for. I went to the place where the trials are, and got two more gems from one of the bandits that you have to kill when you first get there. OK, so now I have three, and went through the first two doors that require the equivalent number of gems, etc. How many more will I need to find, and where are they? I've already done pretty much everything there is to do on Solstheim, and I came across this little side quest after I've already covered a lotta ground that I don't feel like re-treading. Any bit of guidance would be greatly appreciated.

Revves Sarvarni the general goods trader near Tel Mithryn will sell you one and you only need 4 to complete the puzzle. Incidently I believe there are actually 5 available in the DLC.

Theres like 5

Do you mean there are 5 or a number like 5 ?

There are 5 gems total

1. Sold by Revus Sarvani near Tel Mithryn
2. The Reaver Lord in Kagrumez itself has 2 of them
3. There is one in Fahibtharz
4. There is one at the end of the Nchardak Aqueduct when you do the quest for Neloth