I have had very few people actually look at my entire library, or even have access to my mp3 player on shuffle all. I have discovered, however, that my own ideas about my music are very different from other people's.
For example I have a Veggie Tales song, http://www.youtube.com/watch?v=XaWU1CmrJNc, on there. To me this is a fairly innocuous and fun little song from my childhood, nestled in the Sea Shanties folder next to The Good Ship Venus and Barrett's Privateers. I wouldn't ever think someone would be freaked out by it.
But lo and behold it's not the Anime music, or the Jazz, or the JPop music, or even the http://www.hongfire.com/cg/data/25/caramelldansen.swf that gets people. It's Veggie Tales.

Edit: So I relegate pretty much 100% of my library for that reason as "not for other ears". People
can't handle the music.
