I have a scripted spell that I want NPCs to register as unpleasant, but not outright hostile on the first cast. If I tick the "Hostile" flag on the magic effect, then it is regarded to be outright hostile even if it does no damage of any kind - the struck NPC immediately attacks me, as it should.

I'd like it to lower the disposition of the struck NPC instead - so on the first cast, it would make the NPC unfriendly to the player but not hostile. However if the spell was re-cast in a rapid succession, it would force NPC into "enemy of the player" disposition and cause an attack. How can I accomplish that with Papyrus? I can't seem to find any functions that manipulate the NPC disposition to the player.

You could certainly 'fake' the latter by simply counting how many times the spell has been cast on a specific target (and even in what time-frame) and have them attack you after that.

Not sure on disposition but perhaps you could add them to a special faction and each time you cast it on them their rank increases (and so does their enmity with the player). I'm pretty sure factions are quite good at handling disposition.

On the repeated casts, there are plenty of ways to deal with it. You could add a "debuff" magiceffect with a given duration and a proper keyword, and add a blocking condition on your spell so that it is not cast when the target has this keyword. Or you could just use Utility.GetCurrentRealTime(), store that result somewhere and check against it the next time your effect is applied.

Now on the disposition/reaction... There is no such built-in mechanic in Skyrim, and factions's reactions are only a number between 0-3 that influences combat only. No personal factor comes into play to determine the merchants' prices, the NPC's mood when they talk to you, the persuasion chance, etcetera.

So you rather have to ask yourself what you want to influence, then devise a way to achieve this.

Actually, I believe there are a quite a lot of vanilla dialogues, like greetings, that are conditioned by http://www.creationkit.com/GetRelationshipRank which also have the voice mood set accordingly.
Prices/persuade chance can be easily changed via a perk on the player (no npc edit needed) for each relationship level.
Anyway setting that rank would be what op is looking for if wants to keep the vanilla flavor and the number of levels provided are enough for the task.