Like if your just a plain warrior or rogue type? Or will I have to bust out a destruction spell?

A flame shout will do ; if you are a Nord, the power cry, the one that makes everyone flee in terror, will work too. Unfortunately, we cannot drop lit torches...

shoot an arrow at a hanging lantern

Yeah I was hoping an arrow would work on already spilled oil, like in a sloppy bandit lair but no dice. Ok, i'll look closer for LIT lanterns

Thanks all.