I'd turn the compass off for the same reason as you say, for the joy of discovery in exploration, but it annoys me that entrances to some dungeons are nothing more than a darkening in an overhanging rock. I could literally walk right by it and not even know there was a dungeon there if I didn't mouse over the entrance and it said something like "Somekindof Pass". If the dungeon entrances were more recognizable like they were in previous TES games then I'd gladly turn off the compass.
Also, why turn off quest markers? The game doesn't do even a decent job of pointing you in the right direction. Most quests I've gotten so far just said "Go kill the leader of "SoAndSo Cave", and I would have to look at the map to figure out what cave they are talking about and where is it. And with lots of icons on the map that in itself can be quite annoying if the cave isn't in the vicinity of the quest giver (for example the Jarl in Falkreath makes you travel to caves that are practically in the vicinity of other towns).
I haven't had trouble finding cave entrances. Usually there is a narrow dirt road leading up to it, or else there is a flapping banner on a stack of stones nearby. These are both reliable indicators of something interesting, as are things like ruined stone, bones, and other debris. Sometimes there is a sentry posted near the entrance too. If anything, it's much easier to find cave entrances in Skyrim than it is to find them in Oblivion, since there's no waist-high grass obstructing your view.
For the random quest assignments ('Go kill so-and-so at this-and-that cavern"), I just find them on my map. If someone were to tell me to go to a particular place, I'd expect them to mark the location on my map — and the game does this, so I can handwave it away. Some quests may require you to find stuff like objects or NPCs in a given location, but this just means that I actually have to look for it, instead of just following the marker.