The North (north of the neck and the Trident) is the largest province in Westeros. I can't remember exactly where, but it is stated that the lands that Eddard ruled over is more land than any other fiefdom in Westeros combined. When Robb seceded from the nation of Westeros he technically had the largest nation (landwise) at his command. That included the lands of Riverrun which pledged their loyalty to the king in the north because of the the marriage of Catelyn and Eddard. Where the problem arises, though, is that the Northern lands may be the most expansive, but they are also the most uninhabited. Robb has the most expansive and hardest to conquer lands, but that also means he had one of the least manned nations in Westeros.
IIRC Robb had about 3000 to 4000 men when he took the whispering wood and captured Jaime (a surprise attack, remember), which were only a fraction of his total forces. He left many forces behind in that assault to defend the trident and the kings road north of the neck. If you only count the people who are truly loyal to the North and the Starks I'd say that Robb Stark has at least 30,000 men at his command, but they are not all at the same place at the same time (something that the tv show doesn't convey very well). One of the main reasons Joffrey changes his wedding vows from marrying Sansa to marrying Margaery Tyrell is that he needed the Tyrell numbers to have a larger army than any other in westeros.
EDIT: I was thinking you were talking about Robb Stark, not Baratheon. The thing you have to consider though, is that Eddards army WAS Roberts army. When the war started they were in the same place at the same time, with the same stakes placed on both their heads (literally). The Lannisters did not enter the war until Eddard and Robert had all but taken the throne. Tywin didn't join the war until Rhaegar was dead. That's one of the points of contention between the Starks and the Lannisters. Eddard was always wary of the Lannisters because they did not enter the war until Robert was on the edge of Victory. Remember, the Lannister Bannermen got into king's landing because they lied and said the were coming to HELP Aerys, but sacked the city once the gates were opened to them. Lannister Bannermen, just like Frey bannermen, Didn't fight fight in the war until victory was already basically decided for Roberts side.