Some of them are found in dungeons, so if you don't know better you'll just pick it up.
If he lets the red crosshairs dictate in-game morality, then yes - taking a gem from a cleared dungeon is not technically stealing. Although, I can guarantee that whomever the OP took the 1st gem from would not have approved of him taking it.
Frankly, I think if you're going to turn your nose to an entire section of the game due to some sense of fantasy ethics, you should at least be consistent and not steal from anyone - regardless of location/hostility.
Just my 2c.
Besides, the reward for finding all of the stones is pretty incredible. I won't pass it up on future characters.