Does anything change, at all, if you don't complete the optional objective to wipe out the Silver-Hand? I've always wiped them out, but it seems to me that it's not really optional since you would probably kill them all going through the dungeon anyway. You actually have to go out of your way to avoid completing it.