So I'm over encumbered and when I approach the Captain to give me safe passage back to Skyrim I enter a load screen and I'm still in Solstheim. Even gold is deducted from me.
However when I'm not over encumbered I travel to Skyrim just fine as if it were quick travel.

Yep, carriage drivers in the regular game do this too. You must be too heavy for the vehicles!

i had this happen. Using Black book fixed problem for me

Book name?

Black Book: Waking Dreams

just noticed the fast travel in map though. Its in the bottom left corner of map