I have the one thing they should fear - a fence that requires a key and is taller than 3 feet.
Unless they bring a follower along, and give that follower a lockpick, then order that follower stand next to the gate, then get 'uncomfortably' close so the follower backs up through the gate, therefore opening the gate.