I only feel bad whenever I kill one of my followers by accident - i.e. an accidental swipe of the ebony sword or too much lightning in the wrong place. Otherwise, I don't do quests I don't want to.
Oh god, don't remind me. 80% of the way through the capture of Windhelm, I accidentally DECAPITATED one of the mob of soldiers they send with you. Total mistake (I think my original target got killed just before I finished a power attack), but it resulted in the full on head-rolling animation and all.
Obviously, the entire squad turned on me and damn near killed me. I had to whip out a pacification scroll while backed into a corner, and even that wasn't enough to stop the (higher level) squad leader from attacking me.
I ran for the palace with this angry squad leader chasing me the whole way. Fortunately he stayed outside when I went in, and finishing the quest reset his attitude.
Totally FUBAR.
And this is why I now only take companions who have long range attacks that they prefer to hang back and use.