After I cast Calm on someone, I need a way to identify if that target was friendly or hostile to the player in the first place, before being affected.

"IsHostileToActor" will always return false, since the target is already pacified after the effect lands, so it's useless in this case. I also tried "GetFactorReaction", but seems that both draugr enemies and civilians in a town are neutral to the player, so this also can't be used.

So, how can I identity after casting Calm if a target is a real enemy (who would attack the player if not calmed) or just neutral/friendly?

The best solution here is to determine the hostility before the target is pacified. You could use a scripted effect that cast the pacification after determining if the target is hostile or not. You could even make the effect cast different spells on them depending if they are hostile or not and then use those spells to identify if they were hostile or not.