Do you have the game installed to the harddrive or is it running from the dvd?
If it is running from the dvd: Do you hear lots of drive activity when performance goes down? Even if there are no scratches on your dvd, there might be a problem with your drive reading it.
Btw.: The xbox 360 might use space on your harddrive for swapping data even if you haven't got the game installed.
How much space is left on your harddrive ?
I noticed that some games slow down when the harddrive is almost full. Duke Nukem Forever for example runs worse the less space there's left on the drive.