So far, I have 4 Unusual Gems, after you collect one, it says to bring the gem to an appraiser. The game shows me that the Appraiser is just outside of Riften, but you see, problem is, I can't even talk to him, he doesn't even look like an Appraiser, more like a Warrior type guy. When I try to talk to him, he says to PISS OFF! or "You're not worth my time, get out of here!".
What do I do? Or is it a bug that the game has appointed me to the wrong person?

Did you say anything like "You don't scare me"? If so, you can't talk to him anymore. The only way to get it appraised is to join the Thieves Guild.

No, I haven't said anything like that all. I didn't have the option to talk to him at all, not even the first time I met him.
And the Thieves Guild? Really?

Isn't the appraiser Vex in the thieves guild? Go talk to her.

u need to do the thieves quests

Yeah, if you don't/can't join the thieves guild, that guy's the only appraiser you're told about......which is pretty "teh svck" considering he doesn't talk to you.

tl;dr:
I had the same problem with that guy on my character who couldn't join the thieves guild