But you can't physically see how strong or agile your character is, at least not beyond what is painfully obvious, which is that your average mage can jump just as high and run just as fast as a battle-hardened warrior. Maybe that makes sense to you, and maybe you don't even want it to make sense, as long as you can avoid numbers and having to deal with other complications than picking completely meaningless perks in order to somehow describe the character you're trying to create, but I hope you'll pardon me for vehemently disagreeing with that approach.
You can see agility and physical strength, and yes it's painfully obvious. Madden is an example of a game that uses numbers to represent abilities of the players and it can be highly addictive to watch your players progress over time slowly gaining elusiveness, strength, etc. But it doesn't make them feel natural, I never relate a number to speed unless it is in a video game. As for your mage/warrior thing that is true even with attributes. I can make a mage who is just as fast, of not faster than a warrior in Morrowind, attributes don't inhibit that at all. Does it help to classify your characters more easily? Yes, but at the same time your warrior who is specialized in 1h will do more damage than a mage who doesn't specialize in it (via perks). Both arrive at the same alternative just through different means.
And to answer your question a solution would be for weight to make a difference on your characters speed, and jumping ability. If your actual muscle mass affected your damage, and the weight of yourself and your armor slowed you down then you would have a much more organic, natural alternative to reach the same destination as if you had just plain attributes.
You seem to be missing the point of stats. They allow for any number of thing (i.e. damage, lock picking, anything you can imagine). They are not just there for show, despite what you seem to think.
I understand the infinite depth of numbers, and how they can be applied to any situation. However for me I prefer a game with less numbers, not because I don't like numbers, but because for me they bore me, and distance me from my character. I find it hard to relate to my character if he is just the sum of his numbers.