You're in Skyrim. Undead are all around and the events of Oblivion are still discussed here and there. The house is abandoned, and no one has been seen going in or out. You're not trespassing as much as you're investigating, much like how you "broke" into the house in Morthal. It doesn't affect your moral standing in the least. Oh, and let's not forget what Markarth
really is, either. I doubt the guards, or anyone else would have helped even if the Vigilant had proof that something foul was going on. Maybe you're a master illusionist too, and for whatever reason you thought you would be able to reason with the Vigilant inside if he were really going to kill innocents (but lets also not forget that "innocents" in this game tend to attack you, a lot).
Being trapped in the house and forced to kill someone you had just decided to help is a problem a lot of people have with this quest already, but since Molag Bal is a powerful Daedric prince, there's really nothing much we can do. We have to kill him to defend ourselves, which, again, does not affect our moral standing.
Molag Bal mentions his intentions for a priest of Boethiah, and tasks you with
saving him (as for his other motives.. lets find this guy and warn him). Even if you didn't actually want to have him confront Molag Bal, you'd still want to go out of your way to safe another life, no matter how corrupt. Maybe it could be seen as a penance for having killed the Vigilant? So you go off to find and save this man. However, upon finding and saving him, the only dialogue option you have is to tell him of Molag Bal, which causes him to rush off to Markarth. Now, I don't know about you, but when someone is about to jump out of a building or step in front of a bus, I'd like to think I would try and stop them. This is no different. So, you go back to the house to try and make sure the priest doesn't get killed (of course, you were just in that house and forced to kill the Vigilant, but lets not forget he attacked you and there actually may have been a way out -- either way, you may not be thinking rationally and simply wish to do anything you could possibly do to make sure it doesn't end badly.. again). [censored], now you're trapped once again and tasked with killing a man, innocent in your point of view, by beating him with a rusty mace. Yay.
There are so many things Bethesda could have done with this. You should have been able to warn the priest in such a way that he did not wish to go back to the house. You could have desecrated the altar yourself. You could have broken the "cage" the man was in and tried to flee (you're the [censored] Dragonborn). You could have "locked" the house after leaving initially. Hell, you could have done
a lot to prevent all that from happening, but Bethesda simply did not give you the tools to do so. They did in other quests, such as the one from Dawnstar, so it's not as though Bethesda was incapable of doing it. They simply chose not to, and that was a mistake on their part.
Confining our actions to what's possible in-game is completely moot. The entire point of this thread (as I see it) is to discuss what Bethesda could have done differently, which is to say.. a lot.
Oh, and before anyone chimes in with, "He's a Daedric prince! You have to do what he says, or else!" - we stood up to and
demanded Clavicus Vile take back his dog, pissed off Azura, ignored both Vaermina and Hermaeus Mora, and sided against both Hircine and Mehrunes Dagon. Why the hell can we not tell Molag Bal to lick our boots as well?