I am guessing if you're using 2-3 followers its fairly tough to balance the game since enemies have a lot more targets to pick from? I just find that my followers seem to do an insanely good job tanking enemies that can devastate me in a few blows. I am playing on expert with Asis, Deadly Combat, Skyrim Immersive Creatures and Sands of Time; I am wary of playing on Master though since I am playing a character that's fairly fragile and enemies can bring me down quickly.

You could try a harder encounter zone file (where enemies are considerably higher than your level). Some followers at start are level 10 (I think the hirelings) so they may overpower some low level encounters. ASIS encounter zones might still not be good enough, find something more hardcoe.