If you want to get out and about without running the risk of messing too much stuff up, do some generic quests/missions for jarl stewards where they assign you bounties/rewards, and find out about caves through rumors and that sort of thing. That should set you up and make sure that you don't bug anything.
After I got comfortable with my characters and their survivability against the potentially deadly unknown, on my way to missions I'd usually run into whatever caves/crypts I could find. It was pretty fun. Only on occasion did I have a quest to clear it out, in which case I'd just go back, take down one guy, and be done with it.
Also, as for glitching quests.... in general, I'd say that if you find an item that seems "special", hold on to it.
Can't really give many criteria for this. After playing Elder Scrolls for a while, you'll just know what you should probably hold on to

sometimes you can't drop stuff because it's quest related, so there's that. But other than that, if it has a unique name/stats/other properties. Hold on to it. If it seems especially expensive or otherwise out of the ordinary... hold on to it.
You'll find enough loot on the side that a couple hundred gold here or there won't hurt.