Iv been under the assumption that they just a means to perk points, is this the case or does your effectiveness get better even if you don't pick perks?

Your effectiveness won't improve, yet the world will improve around you. Makes for quite the challenge.

Skill point are important.

100 skill points give you

Weapon skill (1-handed/2-handed/bow) + 50% dmg
Magic school (all) -40% magic cost
Armor (light/heavy) + 40% armor

Bingo.

If I remember correct, skill level does increase skill proficiency, but by a pathetic amount when compared to perks. 1 to 100, iirc, increases proficiency by about 50%, which is a drop in the bucket when you consider perks can increase proficiency by 100% and that that 100% is based off the 50% increase to skills.

To explain, let's say a bow starts out with 20 damage. You level your marksman skill to 100 but take no perks in it, the bow will do 30 damage. But if you level it to 100 and take all the perks, then that's 30 damage from skill and then 60 damage thanks to perks. The damage increase from leveling the skill simply cannot compare to the perks.