They built more in Vvardenfell, and tried to do so in Black Marsh with http://www.imperial-library.info/content/argonian-account The difference could be because, I'm not sure how well the opinions of the writers of the 1st Imperial Pocket Guide reflected the view of the Empire in general, but the way they talk about Skyrim is very different from the rest of the provinces, in a really devoted fan kind of way:
"Skyrim, also known as the Old Kingdom or the Fatherland1, was the first region of Tamriel settled by humans: the hardy, brave, warlike Nords, whose descendants still occupy this rugged land, and, although perhaps somewhat reduced from the legendary renown of their forebears of old, the Nords of the pure blood still unquestionably surpass the mixed races in all the manly virtues."
"The Nords of the First Empire never conquered the whole of High Rock; the Cyrodiils ruled it, but failed to stamp out its virulent sectarianism, which sprang up again with renewed fury during the Interregnum. It is only now, under the guidance of the Third Empire, that High Rock is finally tasting the fruits of peace and unity, although a few Bretons still chafe under Tiber Septim's firm hand."
They might've considered the place not in need of their influence as much as the other provinces. Of course, the attitudes of the Imperials now seems to have changed quite a bit with the new dynasty and the Stormcloak war.