set dlc02 to GetModIndex "BrokenHearts.esm" (or just BrokenHearts without the extension, or Broken Hearts with an included space)
returns 255; obviously wrong

set dlc02 to IsModLoaded "BrokenHearts.esm"
returns 0; wrong as well

What do I need to change to get a proper return value (which should be 2 in my case)?

Queue

And if you look carefully at my post, it's evident that I was simply being an idiot. The expansion is called Honest Hearts, not Broken Hearts, so yeah...

GetModIndex "HonestHearts.esm"

is the correct syntax and works as expected. Note that a failure value is 255.

Queue