I'm not sure exactly you mean by "new" books. Do you the new books that were introduced with the Dragonborn DLC? I haven't put duplicate books on any shelves. Regarding only putting what the shelf can hold, usually if you attempt to put more on a shelf that are allowed the game will prompt you saying "you cannot put any more books on this shelf". When you get this message, is this what causes the books and/or the shelves to bug up? Do you have to consciously make sure to count how many books you're putting on a shelf to avoid the bug? I mentioned previously that the books don't stack neatly, but also sometimes some of the books will vanish forever for no apparent reason when I activate the shelf. This seems to cause the count of the number of book that a shelf can hold to go up and obviously this would cause some the above issues.
If your bookshelf has lost its wits, and the books are melding together (sticky books) when you remove the offending books, the number shown for how many books the shelf will hold will increase, usually by double how many faulty books there were. Do not ever put more books on the shelf than it could originally hold, you risk losing some of your books. Isolate the faulty books by themselves, I usually just dumped the lot from that shelf in a container by themselves. You will be safe to put the original number of books back on the shelf. I only ever had the fault with three shelves, which ended up allowing almost double the number of books by the time I finished playing around with them. The fresh books have never bugged out, the sticky books can be read fine in their container. If you just take all the books off the shelf and dump them in a container, you will lose no books, but if you only take the faulty ones off, the problem can spread and you might lose some. When putting the books back on the shelf you would need to manually count them to be sure only the right number were put back.
Often when you take all the books of the shelf, some will still seem to be on the shelf, when you access the shelf again they will vanish. Yet if you check the titles of the books that seemed to have vanished, those books were actually collected the first time, they are still in your inventory. Just do not go throwing those books on the floor, they might vanish, they will be safe put in any container.
By new books I meant those you had collected in Solstheim, you never brought any over from Skyrim with you. If the bug ruins all books of that title I never noticed it, but it is possible. You ruin one sword with it falling through a display case, all the swords of that type are ruined.