Can you? I want to make her my wife, but I don't always want her to wear this bulky, manly armor.

I think she can only wear armours that are better than the one she's usually wearing. Maybe you can find mods somewhere?

I fingered this one out myself.

http://www.gamesas.com/topic/1319527-how-to-get-lydia-to-strip-4-you/