Ok, made a character that focuses on x15 dagger attacks. To do this my sneak skill is at 80 and I have the invisibility spell.

Basically the game can't tell me if I'm hidden or not with my character type.
I'll sneak attack and enemy (they die) his allies who just saw me kill their friend come running to kill me. (this is where things get strange)
The "eye" symbol is now wide open so I go invisible and crouch ("eye" symbol is still wide open) They run straight to my position and raise their weapon... then nothing... they just stop. I stab them in the gut and they die.
Other times: I kill the first guy, friends come over to kill me, I go invisible and crouch, "eye" wide open, Then they apparently really see me and start killing me.
I just don't ever really know if I should take the "eye" seriously.

This will happen every time a foes allies see me and I go invisible... the eye will stay open but half the time its serious and the other half the time it's not as I can get x15 with the eye open.

Sounds like sneaking without invisibility too once you get 70+ it's damn impossible for normal people to find you.
On my assassin I could run up to people and they would look at me then bam x30 sneak attack...

Not that I don't mind if I didn't sneak attack almost everything on that character I would lose the fight.

So the eye stays open yet the sneak attack is valid even without invisibility?

Well the eye just shows if the enemy knows you are around. Even if an enemy does not see you it will still open if he hears you. It probably does not close because he knows you are somewhere close even if he cannot see you. I noticed this when trying to sneak on a quest on my warrior, the eye would open even if guard was out of line of sight, but i realized it was cause he heard me. I took of my heavy armor and i could sneak without issue. So basically the eye indicates if he is aware of you or not, not just if he can see you. So in your scenario he has seen you go invisible so he knows you are still around he just can't find you. So that part is working as intended.

If you are supposed to get sneak attack bonus while invisible is another matter. Dunno if that is intended or not, but well he can't see you so you should still be able to catch him by surprise.