Why do people continue to live in New Orleans, especially after Katrina? Why do people live in LA, both knowing and expecting "The Big One" to come at any day? Why do people still live around active volcanos and in floodplains, worldwide?
I COULD point out that the narrow snippet of rock makes for a very good defensive position. I could also point out that said snippet of rock may not have originally been so precarious when the city was founded. Or I could point out that one of the Jarls of Solitude may have had the Mage's Guild magically reinforce the bridge. But in reality, humans are, as a species, bat**** insane. When you realize that, it'll explain a whole lot about human behavior...
This. Defense.
They also have to run up hill if anyone attacked.
There walls are amazing so that Adds even MORE DEFENCE
And they have the Whole Leigon army.
So really Solitude can't be hit, Unless your the Dovah

Also what Poohunter said