Honestly, I think the weird part about this conversation is the worrying about taking off your armor just to go into town. Then again, that might come from me being a bit more on the barbarian side of things.
It depends on the role of my character.
My thief would wear normal clothes to not look like a thief.
While my barbarian wears his armor while sleeping in his bed.
The character i mentioned on this topic is a warrior kind of type, but also uses restoration magic.
And unlike other warriors, the character is quite fancy with clothing.
It also depends on which town i am in, i would never put away my armor in a small village, orc tribe or a town where i have no home.
And i also keep my weapon equipped, just in case.