But it takes something like twice as high a skill level to do so without the associated perk. At any given level of skill, you can temper armor to a much higher level with the perk than without.
Still not worth it. My current character has 100 in smithing but zero smithing perks. When I temper my gear, I equip a necklace, ring, and gauntlets of Extreme Smithing, and that gets my gear into the "legendary" category. Using a smithing potion on top of that will further increase it's armor rating. Sure, it'd go even higher if I had the associated perks, but once you're in the "legendary" range, you're already gold. Physical enemies can scarcely dent my character and I'm a light-armor wearer.