This happens a bit on my guy too, even without any diseases or poisons on my.
I've even been told by a guard that I prefer bow weapons even tho i have leveled it up about 5 skills points and put no perks in it at all.
Seems to me the ai is a little off about what's going on around them tbh.
Out of curiosity, did you have your bow equipped at the time? For me, same character as before, dual wielding axe and sword, but I had my bow equipped but not drawn when I went into town, and I got the remark about being a bowman.