Mine has to be helgen or windhelm and maybe markarth aswell here's my reasons..

helgen: u start there the place brings u memories of how u first started but as u know when the game starts ur about to be executed..also the place gets to be fully burned down.
Later on when u visit again u have nothing but old bad memories of it.Burned houses..burned corpses and later on bandits make themselvs at home in what it used to be a prosperous village..

Windhelm: I never joined the stormcloacks, never really liked what they stand for, but the main reason i hate windhelm is for it's cold stone like walls and the constant snow falling. Also the overall aspect of the city looks dead and awfull. Another thing is the butcher who killed soo many beautiful ladies!
A place filled with a mad murderer, racist nords who bash alot of the dark elfs and a constant heavy snow which brings in more pain makes windhelm the most unhappy place for me to live in.

Markarth: the city is beautiful but i hate the main storyline about the forsworn.
It bugs the guards, which always makes it harder for me to live in peace on that town.On top of that, i always hated the falmers, bad memories from them.
First time i encountered them, was when i did a mission for the markarth's jarl wizard, who wanted me to gather journals at a ancient dwemer ruin.
Knowing that u get sent to jail at a mine and that there's a giant underground dwemer ruin with a giant spider nearby makes me feel...really uncomfortable..

whats yours?