I play with a mix of HUD. Since i only have 100 health, my health bar is important.
However, i play without any sort of crosshair and i recently disabled the compass and quest-arrow.
There's two mods that make that playable, one that identifies caves as belonging to any major quests and one that gives you better quest descriptions so you can actually rely on the description only to find your quest target.

My game is similar, though I do have still have quest markers on my map, only. So I tend to rely on my map and visual cues for navigation.
And I would not go back at all. Magic can be tricky to aim without crosshairs, but arrows are surprisingly easy. And the badly intrusive compass is gone, which is the best part. When I had it on, I tended to watch the compass more than my surroundings, since most important info would appear there before I ever saw it with my eyes, whether it was a cave or an enemy.