Got this quest by drinking, cleaned up the temple, talked to the farmer with the goat, then to Ysolda in Whiterun, she points me to the Morvunskan fortress, at the quest location at the bottom the portal appears but nothing happens.

Did this on another character and it grabbed me as soon as I got close.

Solved, worked after reload.