This really bothers me. Personally I prefer the Vault Suit and the BOS Officer's Uniform under my combat armor anyway but I'd love it if the fatigues didn't make me look bloated.
What makes it even worse is that for some reason you can only apply ballistic weave to a few under-armor garments. This doesn't even make sense. It's simply a matter of aesthetics. I can wear the fatigues under my combat armor and have the extra 110 damage resistance from the ballistic weave, but I can't wear an outfit under my combat armor that I actually like the look of and get the same benefits? Like I said...the only difference here is aesthetic. I'm not going to have some kind of 'advantage' by wearing an armored BOS Uniform rather than armored fatigues.
It bothered me enough that I added a mod to open up basically all clothing items to ballistic weave.
Here's a full (I think) list of under-armor clothing that leaves ALL armor slots open and can get the ballistic weave -
Army Fatigues
Baseball Uniform
Green Shirt and Combat Boots
Military Fatigues
Minuteman Outfit
Tattered Rags
That is such an annoyingly small list. I really don't get why it's so limited. Like I said, there really is no issue with being overpowered by being able to apply it to everything. I can already apply it to one of these outfits and get the maximum possible benefits of having the ballistic weave under my armor. Why can't I apply it to an outfit that I find aesthetically pleasing?
If you're on PC and not opposed to modding, this is the mod I installed. It lets me apply ballistic weave to almost all articles of clothing, and it also opens up things like the linings for the Vault Suit to some of the other clothing.
http://www.nexusmods.com/fallout4/mods/1860/?