Theoretically a character who can't use magic at all mightn't even be able to use soul gems to recharge a weapon / staff.
I do remember reading the book about white and black souls and the dangers of black soul gems. If touching a black soul gem with the wrong size soul in it could take a piece of your soul into it just by touching it... I suppose an enchanted weapon might be recharged just by touching a soul gem. I'm not 100% on if using a soul gem to imbue something with magic would require the person to use magic but I would think so.