So the quest tells me to go to Serpents Bluff redoubt, which I have already cleared a long time ago.

Anyway, I go in there and it's completely empty. Just a couple traps. I go up the stairs and there is a gate that I can't open. I checked in every little corner with the brightness at maximum, there is no way to open the gate.

I assume that when I cleared it the first time, a gate closes behind you so you can't go back to the entrance, but now I can't get any of the rune weapons because I need the hammer.

Well, I am afraid the quest to get it is probably bugged/glitched. The only thing I can think of that might have a chance of working is too wait for 20-30 in game days to have it reset, but even so I have no idea if it will work.