so whenever i am in a fight with 5+ people VATS doesn't work when i press its button it or if it does it lags so much i get killed.
I've checked the controls and i've changed the button but nothing seems to work Idk what else i should do

This is a common problem, I get it all the time with more than a few people within vats targeting. Best thing I can suggest is grab some sort of automatic weapon, the light machine gun works wonders, and pump it full of armor pen rounds and spread their torso across the desert. I say armor pen rounds cause well you wont be aiming for the head, and most enemies that swarm you like that usually have a decent DT and well worth the slight damage reduction. Or actually use your followers...Boon is hax for clearing swarms of enemies.

I've also experienced this. I just busted out the combat shotgun and kept firing and stimpacking until they were dead. At first I thought it had something to do with some kind of mission challenge...that I was supposed to engage in combat unaided by Pipboy targeting. But no, it's another of 50,000 bugs in this game. Yay.