History doesn't agree with you. Medieval-style shields were strapped across the back using a tight-fitting (because anything loose would hurt like the dikeens) strap that wound through a central bracket, which was only used to hold this strap. The straps were unwound and discarded well in advance of any battle.
You are correct, but I think we need to recognize at least two different ways of, and reasons for, carrying a shield on your back.
1. Across longer distances, like when marching towards a battle far away or traveling across the land, it makes perfect sense to secure your shield tightly to avoid any uncomfort because of the long distances and time you need to wear this huge and bulky thing on your back (since carrying it on your arm would make you tired after a while).
This way of carrying a shield on your back would require some time to get the shield ready, but is more comfortable.
2. When in a more-or-less friendly environment, like in a town where you don't expect to be attacked, or traveling shorter distances, it would make sense to have your shield slung across your back in a loose manner. This serves several purposes - not only does it protect your back from an unexpected attack, but it frees up one of your hands so that you are able to efficiently interact with your surroundings. Try browsing wares in a market with one hand tied up with a shield - it kind of makes it awkward when you're reaching for that coin purse, counting coins and holding the merchandise at the same time.
This way of carrying a shield on your back would not require much time to get the shield ready, but is slightly more uncomfortable since the shield may be moving more when you walk and do other things, but would still be bearable (get it?) over a more or less short amount of time.