Yes... Fast travel is a game-dampener... It needs limits...

"You can't fast-travel to this destination. There is no known safe passage." {For areas too far away, or inland, without roads. Places they want to force you to walk.}

"Your travels have been interrupted." {Drops you onto a path, along the way. Just out of view, but facing the "interruption". Bandits, dragons, a messenger...}

{The feeling of a general sense of safety on the roads... plus having destinations that are not miles-away, would encourage walking... But the game discourages walking, for those who have issues with things like this.}

"You have not found a clear-path to this destination." {For destinations you have found, but have had an encounter along the way.}