I've stumbled upon a few quest items, but I can't put them in one of my containers, and they are starting to weigh me down! Any solution other than complete the (unknown) quest?

You can't store them nor drop them.
However, the plus side is that they don't weigh a thing, despite what the description says.

Oh, really? That makes it better ,then. Thanks.