Is there any benefit to levelling skills without plumping perks into them?

EG For bows does your damage or speed increase incrementally as your bow skill goes up, or does that only come from adding damage or speed perks?