Say you were playing a NO-PERK character, as in not getting any perks period.

Would someone with 50 skill in single handed weapons be better then someone with 25?

Or are skill levels only there to unlock perks?

It does something for smithing (unlocking higher item improvement levels), at any rate. Also, the predicted value of the enchanted iron daggers I'm making (with absorb health and a petty soul) is dropping every time I gain a skill point in enchantment. Woo hoo?

Not sure beyond that.

The combat iks a complex formula involving perks, skilll levels and character level. If you were level 30 and had a skill of 25 you would still handily beat someone who was level 10 with skill of 100. However if you were within a couple of levels he would be dominating you.

so if you do a perkless build; someone with higher skill will do more damage?

Improving a weapon skill will improve base damage, improving an armor skill will improve the armor rating, spells will cost less, etc.