It's impossible to balance. Are you going to scale it to the player who played the game legitimately and has all the best gear? Or the person who abused alchemy and crafted 9999999999999999999999999 damage weapons? Because both are possible, but only one I have any respect for.
Also good luck to you, sounds like a hell of a task.
There are limits to the abuses in this game. Not only do the loops to improve equipment max out due to diminishing returns, but armor beyond a certain point is meaningless. The premise of this mod is to challenge those that intentionally abuse the system, but due to system limitations those that don't intentionally abuse the system also stand a chance if they try hard enough. If I have any say in this mod the enemies will follow the same rules the player does. As such most NPC's probably won't more than 1000 hit points, though their armor is likely to be maxed out with at least one resistance high enough to make them immune to at least one element. I have a feeling fighting most enemies that are intended to be a challenge would be like a game of Rock Paper Scissors, where the players, no matter how badass, will find that enemies are basically immune to anytime but what they don't have proper resistances for, and the "weakness" will probably kill the enemy fairly quickly.
That said I think challenges that don't involve combat stats will be a big thing, puzzles, stuff that requires intelligence, situational awareness, and reflexes will hopefully be what makes it great.
Edit:
Personally, I don't like when I have to fight a boss as well as others at the same time. I prefer to have to fight through a dungeon getting progressively harder, and then the boss and I duke it out alone in the final room.
Part of the premise of this is that you are so powerful that most enemies are just scenery, Decorations, and a distraction for those that can't focus and see the true goal. You may be dueling the boss while an army swarms you, and dragons rain fire down upon you, but that army and those dragons are just there to look cool and tell a story, the only person you are really fighting is the boss the army and dragons can't do enough damage for you to even notice and will die by the score should you even glance at them.