Not always the case. Unusual gems weigh .5 each, and you have to collect 24 of them. The tablet from the first dungeon weighs 25. There are a couple of instruments that weigh 2 - 4. There are plenty more cases.
Once the quest is started for the gems, then they weigh nothing. Only if you collect them before the quest, do they have weight. There are about 4 of these that you want to get the first time you see them, the rest can wait until the quest is active. This is handled in the game as the name of the item changes during the quest. I would imagine it is the same for the instruments. However the Bards College was one of the first places I did quests for and those instruments do not have weight during the quest.