This is just a problem with having a perk system like this, skill seems to mean nothing and everything is tied into the perks.
That's precisely the point and why it works so well. If it were all tied to the skill rating itself, you could become a god in everything... no fun. The perk trees force you to either go the "jack of all trades, master of none" route, or specialize, regardless of your ratings.
I think it's great the way it is... if anything, I'd rather see them add more perks, plus a way to redistribute your perks... but making it possible to max out every perk tree? No thanks.