Yes, you are screwed. Unfortunately, this is the single most dangerous place on the map because it's not actually connected by sea to the mainland. It seems to be a separate space, something like
So you can't swim back.
Also, you can't fast travel from here because the quest that takes you here is designed to force you to take the boat back, so that you don't miss out on your reward. Only problem is, if you ever come back, there's no boat.
So you are doomed. I'm pretty sure this is very much a terrible bug, and Beth will have to fix it eventually.