I think you could use the event http://www.creationkit.com/OnCombatStateChanged_-_Actor to detect if the player enters combat with the people around them. If it's a controlled environment with limited contact, you could place a script on the NPC's. If it's an open environment, you could perhaps place it on the player via alias.
Hm, sounds like a reasonable idea, but yeah it would only effect quests in a closed environment.
Essentially,, my intentions are to set up a city.designed around stealing, along with side areas that have specific stealing quests. In the open city getting into conflicts will not affect the quest, while inside the quest areas; it will heavily change the outcome of the quest if you are too obvious.