It's the opposite, really. Any change is for the worse. Someone using fast travel does everything you do and has exactly the same experiences. He runs into a husband and wife along the road and gets thanked for helping them out. He saves a peddler from bandits, or helps kill him. He skirts cave bears and sabre cats, and runs for his life, or fights, or dies. He learns what ruins and caves and passes are where, and what guards them, and where the sun rises and sets. He learns where streams begin, in which direction they flow, and where they end. He gets to know the pools and the hills and the fallen trees that tell him he is nearing his goal. Avoiding fast travel gains you nothing he doesn't already have. Using it gives him a brief respite when he is ready for something else.
I'm not talking about the people who only use fast-travel every now and then. I'm talking about the ones who deliberately beam themselves from location to location just to finish questlines as quickly as possible, with little or no interest in the world around them.
People like that miss out on a lot of exploration, feel like they're in a game that needs to be completed rather than an immersive and believable fantasy world to live in, and get a very rushed sense of progression through questlines. A lot of the time, such people become fed up with the game within the first 100 hours. Essentially, they treat TES games like story-driven action-adventure games with pretty scenery. They then often moan about the poor storyline and lack of things to do.
The whole experience feels so much more fleshed out and real when you don't fast-travel. Instead of playing a 100 hour story-driven action-RPG, you're playing an 800 hour open world RPG.