I now have close to 4000 saves and I am pretty sure they are taking a crap load of space on my HDD. Seeing that I lack the patience to ever go waaaaaaay back in time and load a previously saved file, I see no need to waste valuable HDD space.

With my patience, I think 50 saved games is plenty. Has anyone tried deleting saved games before? Do I just delete them from the saved games directory? And will the game screw up when it realises save1 to save 3950 are missing and only save3951 to save3999 are around?

What system are you talking about?

I've deleted plenty of save files from my PS3 without any trouble in any game.

No, you won't have any trouble after you delete all your old saves. You might even find that your game doesn't lag as much. Just make sure you keep enough of your old ones to where if you need to revert you have a few decent ones to go to for each of your characters.

I would recommend making duplicate saves of all your characters beforehand so just in case you accidentally delete or maybe even save over one of your characters on accident you have a back up. That happens a lot more than you may think.

My Documents/My Games/Game Name/Saves (Skyrim here for Game Name)

I collect tons of saves and then delete the excess ones later. I usually keep milestone saves and early saves so I can avoid the tutorials.