I don't know if this is replicable, but I may have found a way to stop Brynjolf from inviting you into the theives guild.

I just went to Riften and when I got to the market place I decided to just run up to Brynjolf and get the conversation over with as fast as possible. What was interesting is that I ran up to him so fast I miscalculated my position and ran right into him which caused him to back up a couple of steps rather than initiate dialogue. And he never did initiate dialogue. He just went back to peddling his Falmer Blood Elixir. Don't know if he will try again the next time I go to Riften, but so far, running in and out of various buildings he has left me alone.