I found the solution, What I did was commit another crime, yield to a guard and choose the option to "Pay the fine". That seemed to reset the guards' speech and they wouldn't just say "Wait!.... I know you" anymore when I came into their range. I talked to the guard nearby and the option appeared to pay the prisoner's fine. Finally got to get that quest done!
On the other hand the NPC I had to kill was pretty tough and surrounded by like 5 dudes. Not to mention I found, two giants, a sabre and a mammoth while traveling which were really tough for my level! Run! Run! Run!