Quest items have no weight, so they're not hurting anything there.
That is true, although it's kind of annoying to have them cluttering up my inventory once I no longer need them.
If an item is considered a quest item, it's probably supposed to be, the problem is that some items flagged as quest items are ones that might still be left in your inventory after the completion of the quest. Whether because you chose an option that would cause them to not be lost during the quest or you can acquire more than one of them. One example of the former would be the dog brains you can find in the game. You only need one, but there are three to choose from, if you find them all, the game still only removes the one you chose to use, thus, after completing the quest, you're left with two weightless dog brains stuck in your inventory. The latter would be things like the Repconn rocket models, of which there are many more in the game than you'll ever need, and yet since they're considered quest items, you can't drop them once you've picked them up, thus you're stuck with them.
So I'd say the solution would be to either ensure that all quest items are removed once they are no longer needed, even if you did not use them, or have their quest item status removed once they are no longer needed, assuming they were not removed for any reason. I'd say the latter option would be preferable in quests that would allow it (I.e. quests where it would potentially be possible for some of the quest items involved to still remain in the player's inventory after completing the quest.) this way, you wouldn't have to worry about unintentionally dropping or selling important quest items, but once the quest is completed, you would be able to do as you please with them.