Hopefully i am missing something here and being really stupid. Can you remove the custom marker (Light Blue) on the map?
The reason i ask is the Clairvoyance spell seemed to be pointing me towards my custom marker and NOT the quest marker when i was lost in the depths. Took me forever to get out because of this.
Go back to your map, and check the key at the bottom that tells you what button does what. I think the same button that places the marker also removes it.