Is it just me, or is there basically no difference between wearing light and heavy armor, or for that matter no armor, in your speed?

Its you, I can certainly notice by heavy armored Nord runs a lot slower then my Breton assassin until I get unencumbered.

Do you use the steed stone?