brush gun, you do almost 200 damage a shot, and even with 3x DT on enemies, thats only around a max of 60, so your gaining a bonus of 20 a shot

Damage bonus is applied to damage that leaks through DT. If the gun does 75 basic damage and you shoot at somebody with 15 DT, then you do 75-15 = 60 damage. If you use hollow points, their DT becomes 15*3 = 45, 75-45 = 30 damage goes through the DT and is multiplied by 1.75 = 52.5 damage delivered.
The Brush Gun's raw power is enough to make hollow points advantageous up to target DT of 13: 75-13 = 62 for standard ammo, versus (75-13*3)*1.75 = 63 for hollow points. After that, standard ammo is better.
I'm pretty sure such bullets don't exist, especially in real life. the point of such small rounds is penetration (I'm pretty sure) if you made such a tiny bullet hollowpoint i don't think it would work.
The game's 5mm round is, probably, based off the obsolete 5mm Remington Rimfire Magnum which was a very small bottlenecked round aimed at larger varmints. The real life round has plenty of power to cause effective deformation of hollow point bullets at short ranges. Reportedly will kill coyotes and hares and that sort of thing no problem. It isn't quite as powerful as the PDW rounds like 5.7x28mm that you are thinking of, but the real difference is the bullet design. PDW rounds are designed to penetrate. It isn't a function of their size, it's a function of the composition of the bullet: armor piercing rounds will typically have a dart-like "penetrator" made of hardened steel or even tungsten. When the bullet hits a target, the jacket peels back, allowing the penetrator to (ideally) continue onward through the material. Bullets for hunting are nothing like this, they are made only of soft lead that will deform easily and cause a large wound in the target.