I play on master difficulty, bandits don't fill them. So what do I do?
Here is a short guide
Humans fill - Black soul gems
Mammoths - Grand soul gems
Giants - Greater soul gems
Bears, saber cats, trolls - Common soul gems
Spiders, undead skeletons, skeevers, mudcrabs, deer, smaller animals - Lesser and Petty soul gems