lol I was doing the Imperial quest where you have to go talk with the Jarl in Riften. Her assistant takes you into the room to talk about a deal.

Anyway before that, I accidentally stole a silver necklace from the display case before getting the incriminating letter.

After I talked to the Jarl's assistant I put the necklace back in the case and closed it (since I can't sell it anyway). Once I turned the corner some random dude comes up and says "hey you dropped that armor, could I have it?" I said yes and he took it out of the display case.

Thief! I should tell the guards

yea some npc's ask you that when you drop things.