Quick, we caught another one - haul him into the boat! 
"Drives my wife and me crazy" is correct (apart from the missing subject). "me" is an object pronoun, and since it (along with "my wife") is an object of the sentence, it works.
How can you test this? Take away the other object. You're left with "Drives... me crazy." You wouldn't say "Drives... I crazy."
Edit: gar! Beaten to the punch.

"Drives my wife and me crazy" is correct (apart from the missing subject). "me" is an object pronoun, and since it (along with "my wife") is an object of the sentence, it works.
How can you test this? Take away the other object. You're left with "Drives... me crazy." You wouldn't say "Drives... I crazy."
Edit: gar! Beaten to the punch.
ninja'd you! LOL.