I never allowed it to let him get away.
Speaking with Swank, at the front desk by the greeter, before you confront Benny opens up an option to confront Benny fully armed once you complete an investigation for Swank.
Once done, swank returns your weapons, and they are no longer taken away in the Topps.
Time to confront Benny now, loaded for bear.