http://www.escapistmagazine.com/forums/read/9.323041-Skyrim-on-PC-disable-Xbox-controller-get-a-significant-performance-boost-easier-menu-navigation

I've tested this as well, turn off the 360 controller option in the menu and I get better fps. I turn it back on and significant fps drops occur. I bought a 360 wired controller last night use it for Skyrim, however if not using it will give me better performance than so be it for now. I would like to see the modding community or Bethesda release a patch that fixes this issue were people like me playing with the 360 controller can have the same performance increase.

Thanks to anybody who can make this happen! (I cannot as I have no modding experience)

I can confirm this bug exists, the performance jump I got after disabling 360 controller support is unbelievable! This is definitely a bug that needs to be addressed!