Playing a Warrior, I don't invest in anything but Health, at each level up. I don't even invest any perks in armor, at all. That goes for any of my characters. I may drop one or two in Juggernaut if I am having issuers. Other than that, I don't worry about it.
Sword and Shield, along with Archery, can spread out the skills too much for me. This will cause a faster leveling of the character.
My problem is the defensive skills keeping up with the offensive ones. And if I don't invest in some Stamina (220 Health, 150 Stamina, 100 Magicka) then I won't be able to Power Bash or anything. And if I don't invest points in Heavy Armor to get the perk that keeps it from affecting Stamina use then its pointless for me to use Heavy Armor.
Anyway, the enemies that I had to drop the difficulty down for we're impossible to back out of. I was trapped in the room with fighting them being the only way out. But I have been focusing on my combat skills alot, as well as gathering good equipment and weapons. At level 13 I obtained Spellbreaker, and that helped alot. I'm working on Crossbows and Dawnbreaker as well.
Lidiya is my follower right now, but honestly, its kinda insulting that I need to rely on her just to play on this difficulty. If I'm going to have a follower its for tagging along with. Not to mention followers in Skyrim are a pain in the butt..
I'm pretty much going to drop the difficulty. I'm going to play a pure mage on Master. Or I might just forget about it and only play it on my Dark Elf. The issue is playing it from level 1.