Is there any way of dropping quest items? I'm carrying around some stuff I don't want yet and it's beginning to stack up weightwise.

No way to drop them unless you complete the quest associated with it. And to my knowledge quest items have 0 weight even though there is a weight displayed for the item

This.

I've been able to test that with a few things and it's been true. So far I've been too lazy to drop/store everything I'm allowed and check my weight of 0, but others have and verify that undroppable stuff weighs 0 regardless of the weight on the Description Card.

Or via console, type player.drop . For example, player.drop 0001AD0C 1 will drop one certain book of the Fetch Me That Book! quest from the College.

Great! Thanks for the info everybody. I have 2 of one item and not being able to lose one annoys me. Good to know about the weight as well.
Oh well, time to get back to squishing dragons

If it's 1 of the 2 quest items of which I'm thinking you'll be able to get rid of the extra after you do the quest.

if the quest item happens to be a weapon you can equip it and then store it on a weapon rack in any of your purchased homes. i've done this with a few swords that i don't feel the need to return

You can always go to jail and have all your gear taken from you. I did this, broke and just left all my stuff because I didn't want it. Problem is when you finally pay off your bounty you get it all back.

For pc users, there's a mod which removes the quest tag so you can drop, sell, or store any of the quest items.