When I first installed the game I had this issue. Basically, when you perform one scroll with the scroll wheel nothing happens, then you do it again, nothing happens, and then on this third it goes very fast. For example, you are in first person, you scroll twice, nothing, and on the third it jumps to the furthest you can get in third person in mere 2 seconds. I reinstalled the game and then it worked properly. Today I came home, started up Skyrim, and there is the issue again. I've tried everything, from .ini tweaks to updating drivers. Any ideas ? I'm using a Razer Naga.
P.S: I had the same issue with New Vegas, but I played for like 2 hours, then quit the game for 2 months and when I came back to it, it was gone and never appeared again.