There will be that difference anyway due to the nature of the items. A crap poison makes sense, as does a stronger poison, but making a crap Daedric sword doesn't make any sense at all. You get more xp. for stronger potions after you have invested, you get more xp. for better weapons after you have invested. Of course this is complicated by after the fact smithing improvements, but you can still improve Daedric to 'fine' with no skill and no perks, thus smithing gives you the chance of a small xp. boost over that which alchemy offers.
If there were more to smithing than "Insert components, receive weapon," it could work. Smithing's a tough process, and there's plenty that could go wrong. Your sword might look like your standard Daedric Sword, but it could actually be really weak if you didn't temper it properly.
Though I think too many people would object if the Smithing process was a minigame like what I'd like to see. Damn instant gratification seekers.