How much harder does the game become, if you raise alot of minor secondary skills very very early in the game? before raising your main weapon / defence skills?

My sword / block / armor skills are only around 30, and my player level is already up to 11 (Just by casual play, stopping to blacksmith for a while, alchemy a bit, etc etc).

Im always afraid to level too many minor skills.
I have a few questions about the game. maybe im wrong.

My character is 1h Sword / Shield based.
I was level three or four and then got a few levels from blacksmithing and a few levels from other things that I forgot.
Point is.. I didnt focus on leveling my Sword, armor and blocking skills early. I was level 11 before I knew it and my Sword and block and armor skills are around 30.
Now even mud crabs hit me for l ike 10-20 percent if I dont block. (Even with steel armor set).
I went in to one of the first bandit caves, some of the bandits were pretty fair fights, but the bandit chief was really OP, I had to run out of the cave two or three or four times to heal up and run back in, which is kinda lame.

I remember my last character was up to around level 16 or 20 or 22, I forget, but I remember most enemies being realtively safe to fight.
I like to play slow and listen to all the dialog and do lots of help and quests in one part of the map before exploring others.

My question is,
Does leveling alot of minor skills (And getting levels from those minor skills, (That dont end up helping much in combat even),
Does that make the game harder?

If I had focused on not raising other skills for the first ten levels, so that I was sure my Sword and Block and armor skills were high, before getting levels form other minor, less usefull junk.

Would the game seem easier?
I remember playing on Expert last time and having no trouble at all.

Im level 11 atm, with 30-30-30 sword block heavy armor skills and Expert is seeming very hard. Even mud crabs are pounding my steel armor.

Im sure with more armor skill, the armor will work better. \

I just wonder if my skills are "Behind" and will make the next 10 levels extra tuff.
I wonder if I should start over and get my 3 main skills up a bit further first.

Or does raising minor skills and doubling your level because your raising lots of less usefull skills not hurt anything?

If you level your character by using non combat skills, then the game will give you higher level enemies. It does not care that you have not buffed up your Defense and Attack skills.

I guess ill be ok if I just focus on offence and defence for the next 5 lvls and nothign else.
I assume / hope / think X_x
Its so easy to level up other junks along the way X_x.

so it would be much wiser to focus on offense and defence abilities for the first 10 levels next time then?
im up to 13 and its still rough X_x.
Its not so bad when its one at a time. but the game likes to throw 2-3 in the mix often enough.

That makes it more exciting for me. I died alot trying to kill the bandit chief yesterday and ive perked up smithing to get dragonplate set. My 2 hand is like 50'and my 1 hand is 40. My shield/block is 29. Its pretty fun atm.