You are not crazy. You can exaggerate the effect by setting the iFPSclamp to 20 FPS in the ini file. (In offline mode or Steam will reset it.) The higher above 20 you go the faster you will move. Works in the opposite direction to. Same thing happens in Oblivion and Fallout 3.
When I go into the saloon, with FPS hitting the 80's (Until I look at someone.) my character speeds up quite a bit compared to just outside were I am getting 35ish FPS.
I'm praying they build, or buy a new engine for future TES/Fallout games. This one is past it's prime and ready for the glue factory so to speak.