I've been looking in caves and buildings and the only torch I've found so far was inside a loot chest. But is there somewhere in skyrim where there is a plentiful supply of torches that I can pick up?

Not really. I'd suggest the Candlelight spell.

The torches are for placing in my house to brighten it up a bit. The lights you build are not very bright.

Broken oar grotto had quite a bit, as well as the East Shipping company place (the thieves guild quest takes you to the secret place).

There is a mine close to the Falkreath house that has loads of torches. I cannot remember its name.

I just picked up 20 torches from the broken oar grotto so cheers for that.

Embershad mine

It might be a bit different but what I mostly do is just pickpocket the torches from guards in random towns and if I feel like it reverse pickpocket them to vendors and buy them back.
I assume you have enough money to do that .

To make it yourself easier, wait until dark so you can actually see which guard is holding torches and which don't .
Of course, you need to have the necessary perks to pickpocket holded items.

Ps: the reason I reverse pickpocket the torches to vendors and buy them back is to remove the stolen tag from it. If you don't mind that, just pickpocketing them from guards might be enough.

Embershard Mine (on your way to Riverwood after the Attack from Helgen) has at least 20 torches in it.

I'm not sure if this is entirely correct, but my father said he left and came back in and all the torches he grabbed had immediately respawned. Worth a try.

While they may not respawn immediately, they do respawn after a month or so. It's where I get my torches.