I heard a variety of things about when and how often... Does he only do it on Solstheim? Does he do it on the mainland too? Does he killsteal *ALL* of them until you kill him? Does he do it only if you have 1 or more unexpended souls?
I cannot peek into the inner workings of the quest that governs it as they have yet to release the script source files, all I can see on it in the CK are the script properties (IE: variables and such). The quest is curiously marked "run once", which would mean it happens only once... but the sound of things indicate otherwise.

So.... observations? Try to be as concise as possible, nebulous ill-defined data isn't of much use.
Also... Does he only comment if you've obliterated Alduin? Or does he also have comments if you just sent Aldy fleeing like a sissy from the Monahven?