I expect that you don't know that you can go lower in the list when recharging? Easy mistake as they don't have an arrow indicating the list is longer than 3 options. Hell, I never knew that there were more than 3 options in the dialog boxes when creating a Fallout: New Vegas character. Untill about 6 months later.
Now if you're talking soul gem fragments, those are truly worthless; I don't even pick them up. Just pretty colors to make you go, "what was that??"
I always pick them up, and place them into the little tray-thing next to the Arcane Enchanter in Proudspire Manor. Gives the illusion that I do a lot of enchanting there, which I do actually.