At one point, you probably had your follower pick up this sword, instead of giving it to him via the dialogue functions. Doing this made the sword his property.
I would hope that's the case, otherwise I hate to think that picking up something your follower drops counts as theft when it's something you gave to him yourself.
But yeah, OP, if you know you stole it then it shouldn't be a surprise that he's unhappy about you taking it. And IIRC committing a crime against an Orc will get you in trouble with their strongholds so that's where the bounty is coming from.