I made a few test scripts to try things out.
I can delete them from my scripts folder, and remove them from items, but they still stay in the creation kit's script manager.

is it possible to delete them or remove them altogether?

When the PEX files are gone, they'll stop showing up I'd imagine.