That's only the half of it. It allowed for better armor detail as well.
While you aren't 100% wrong, you are more wrong than you're right. The most important reason bethesda removed pants from the game was a few armor's design not having pants at all. While some clipping issues might've been fixed because of the limited amount of customization, it didn't add much to the performance of the game. Besides, the only time you'll see enough NPCs on the screen for it to matter is the civil war quest. Oblivion probably had almost as many NPCs on screen during the great gate quest.
I'd prefer more NPCs in the game world, but it feels like Skyrim has just as few as Oblivion, meaning, the removal of pants wasn't a deal I would've made.
Just wait until TES VI gives us armor as one piece that covers all previous slots, chest, head, feet, and arms.
Like in Fallout 3 and new vegas, but just the helmet also slammed into the mix.