I don't need to invest any into Speech
Really? You're going to neglect speech, of all things?
Anyways, in my current playthrough i maxed out barter, energy weapons, guns, lockpick, medicine, repair, science, sneak, speech and i have 60 or so in explosives, and i didn't even use every book.
Kind of takes out the fun of character specialization if you ask me, but if you want to do it, get 9 int and the int implant as soon as you can, get at least 9 luck (implant and intense training), figure out which SPECIAL attribute affects which skills, take Educated, Comprehension and Tag!, and track down the skill books.
Taking Good Natured would help too, but that's assuming you have more important "peaceful" skills than combat ones.