I don't want to do the TG questline but I do want the ability to bribe guards without perks. If I complete the first mission to join then will I get that power.

You can bribe without perks by being a Thane of the city. Up to a point of course.

You can use the "Thane card" to make the guards overlook a bounty up to 3000 gold, but only once. After that you have to pay up.

TG members have the option to bribe guards but for small crimes the amount required to bribe them is usually more than just paying your bounty. And it may only work for the crime you just got caught committing, but not clear your entire bounty for the hold if you have committed other crimes there in the past. I had this happen to me, I bribed a guard to overlook something I'd just done but then was stopped by another later on and ended up having to pay my entire bounty anyway.

I believe you have to have completed the "Special Jobs" for the cities to bribe in the respective holds.

Edit:
I believe that is correct.

You know about the barrel in Dragonsreach that clears the tag, right?

They confiscate stolen items if you choose to pay fine, right? I have many stolen items that are part of my essential gear.

I wondered about that, I checked UESP but couldn't find anything that specified it. It just refers to being a TG member, not having done particular quests to get additional influence in the various cities and holds. Maybe the options get better after the special jobs for a city?

I only did the TG once and I honestly can't remember for sure if i bribed a guard before completing the City Influence quests, but I think the incident I referred to in my first reply was before doing them. I was really disappointed that it didn't clear my entire bounty.

What?!?1? There's something you don't know? I better check the temperature in hell. Luv ya!

I don't recall seeing the "bribe" option early on in the TG questline and assumed that it was part of the "influence".