I'm thinking of using it for poison for my illusion assasin.

I use archery, illusion, one handed and sneak obviously.

Is it worth putting a few perks Into it or not?

yes the poisons you can make are insane

If anything, alchemy is a bit overpowered. Definitely worth it if you want to make powerful poisons.

What perks should I take?
Obviously alchemist, physician.
Poisoner? Concentrated poison sounds useful

Take all five of the alchemist perks, poisoner, and concentrated poison. Take physician if you want better healing pots. Snakeblood helps to protect from poison, but take it only if you are getting poisoned a lot or for RPG purposes.

With the exception of paralysis poisons, I find the other poisons to be underwhelming. But that's just my experience. As for alchemy as a whole, it's very much worth taking the perks. Buying potions is an expensive proposition and the alchemy shops don't always have what you're looking for. For example, you're not going to find a Blacksmith potion that will give you a 65%+ improvement in smithing, but with 4/5 perks you can make them.

The only perks I take are the 5 at the bottom of the tree. With those I can make any potion to suit my needs.

In terms of $, alchemy can be a big moneymaker once you have just the first two perks. With more perks than that it can take in-game weeks to sell off your Damage Magicka and Stamina Regen, Slow, Invisibility, and Paralysis Potions. Right now I'm sitting on about 50 potions worth around 800 septims each. I just don't want to take the time to sell them all.

The only drawback to alchemy is that some of the big money/exp potions are virtully unusable in combat: they have strong effects that both hurt and help the target.

Poisons (straight up damage poisons) are more powerfu then people often think. You won't kill strong enemies with poison alone, but using it a couple of times makes a difference, because essientially its a buff for your weapon. I have a fiery soul trap sword and love to apply a "weakness to fire" poison first when dealing with strong enemies like frost trolls. Makes a difference.

Well it does help, not it wont effect Vampires and 50% resistance for Bosmers, not sure about other races. Yes they can help out alot in most situations and its worth it, but warn it can make the game even boring if you make yourself too powerful.