Personally I think it was a fair battle. Not fair as in no predetermined outcome, it was obvious from the moment the challenge was issued that Torygg would die if he accepted, but fair as in he did nothing that is considered immoral or illegal.
I don't think Ulfric was misusing the Voice, because the Way of the Voice is the philosophy preached by a Nord warleader who lost a war and apparently decided that if he couldn't win a war with the Thu'um, no one should. The Thu'um was given to the Nords so they could fight the dragons. And about Ulfric studying to become a Greybeard? He was a child. Breaking an oath you made as a child with little to no choice isn't something I see as negative here.
As for the result being predetermined I think that is a part of these duels. It is a warrior culture and a leader is supposed to be able to protect himself. His title as High King made Torygg fair game for any of the Jarls, no matter how much more experienced and skilled than him they are.