Yes that is true, and Skyrim has been a part of the Empire for a very long time.
Mind that this was the Septim Empire which had ties all the way to Alessia and a different one from the Mede Empire alltogether.
Still, you cant just say it was yours all along and not be an invader.
Its more complicated than that and ultimately has to do with what the people who live there want.
Skyrim is the birthplace of humans as we know them, and the Empire. It was the FIRST province of the Empire, which was made by Talos.
MANY of the citizens of Skyrim see that. I'd wager there are more Imperial supporters than Stormcloaks, by a fairly large margin. Mainly because Ulfric is literally trying to forcefully conquer anybody that disagrees with him using his army, so he can claim the throne of High Kind and rule with an iron fist that the people he manipulated to follow him bled to give him.
Seriously, the Empire is more than justified with their presence in Skyrim.