It seems like most people go to Whiterun and follow the main quest line right away. I'm not going to do that. First I'm going to go to Riften, do quests for the Jarl and get him to like me enough so I can buy a house. Then I'm going to work on sidequests and such so I can get enough gold to purchase that house and furnish it. After that I'll continue doing the Jarl's quests and see if maybe he'll make me a Thane and give me a Housecarl. (I'm assuming that every Jarl can reward you with a Housecarl if you get far enough into their questlines.) After that, it's off to the Mages Guild. I plan on trying to be a pure Mage character, using staves, daggers, Mage robes or light armor, maybe even just enchanted clothing, with a little bit of lockpicking thrown in if it's true that the open lock spell has been removed from the game.
After that, who knows?
I'll go exploring the rest of the world only after I've been made Thane of Whiterun. Too much important stuff happens before that point to ignore it, and it's a fairly structured way to ensure you have a solid starting point for your character (a few perks, some gold, a house, dragon shouts, etc).