I finally get near the Fort, try to go to Cottonwood Cave and it willnot let me in. I keep clicking "E" but nothing happens

Have your spoken with Cursor Lucullis near the entry gate in Caesar's camp? (or whatever he is named)
He has a dialog which can transport you back.

I have used fast travel to leave the Camp as well, which returns my weapons, btw.
I tested that on purpose.

Yes I have. Now whenever I speak to him I get the same business about him waiting around to take me back when I am done.

I have gone back a few saves to before I meet him to get back to the fort, try the talking again and asee if I can use the dialog text to have him send me

Took me a while to find my way back to the cave, still no luck. Spoken with the right NPC, clicked on the gate with "E", no luck.