Let's say I fast travel once an hour. What am I missing? It doesn't make sense to say that you gain something by never fast traveling.
You probably are not missing anything (although random encounters can be fun). But if I did that I would be missing something. A sense of connection with the past. I would have a "How did I get here?" moment of broken imersion and it would take a while to get back into it.