Well there are a few things I can think of:
- Daedra are considered evil in most of society and you do not wish to use them in front of people that might ruin your name (but necromancy is hardly a suitable alternative). You could RP fear of the vigilantes of Stendarr coming to hunt you down for use of Daedra or something.
- You do not have the willpower to constantly fight with and control Dremora Lords and as such feel that it is better to only use them when you have no other option.
Not the first option, my character wouldn't care. He'd welcome the fight brought on by the Vigilants.
I like the second option, only the strongest in the skill of conjuration can control dremora lords, but no one can control them for long, therefore he can't over use them.