For most games, it decreases load times as transferring data from a hard drive is faster than reading it from a DVD. It also reduces noise and heat as the DVD drive isn't constantly spinning. In my experience, I've found many games simply run better and crash less with the game installed. Bethesda examples for me were both Fallout 3 and New Vegas. Both of those games crashed much less often after I installed the games to the hard drive.
In the case of this game, however, there is apparently a bug that leads to inferior textures being used until the data cache fills. Some players are saying that once the cache fills up (many saying in around an hour), then the textures begin loading correctly again. Bethesda have said this will be fixed in the next patch.
It does take a several minutes to copy the data to the drive. Yes, your saves will still work. They will also continue to work if you remove the game from the hard drive after playing with it installed.
If you're happy with the performance playing from the disc, you may want to continue that way until the patch is released. Then install it and see if it plays better. If so, leave it. If not, remove it. I install most every game I play to the drive but am waiting on the patch for this one.