the whole games based around dragons
I don't care. This is my game now, now Bethesda's. Once the game is in my hands, so is the story. Right now I'm doing something similar to what waldorfTM is doing: I'm playing a non-dragonborn mercenary, just an ordinary person trying to get along.
There are game play and meta-gaming and power-gaming (and purely fun) reasons to want Dragons in the game. And there are roleplaying reason to
not want Dragons in the game. Neither of these choices is "silly."