I'm not saying to count that time towards the time for another skill. I'm saying count the time you spent training a skill towards the perk you want in that same tree. You already did the work, you leveled the skill.
[Perk points represent time]
I disagree. Skill points represent time. Perk points are more like realizations that are reached by being so good at something. If you put in the work to train a skill, you spent your time. Short of making every perk obtainable, which is, yes, unrealistic, having a way to change your perks would be acceptable. When you learn a perk, it's instant. You put no time into it. It was a single decision you made in the skills screen, that may have taken only seconds.
Perks represent time too, because you have to level skills to meet prerequisites to get the perks. Either way, it's an ability you learned over time via practice. Granted, you didn't have to make the decision immediately or even beforehand, but it's still an ability that represents the amount of time you had spent practicing. Being able to reallocate that is just not at all relatable to how people learn things.
My biggest problem with starting a new game is not relearning the skills and perks. Its everything else. Everything that had nothing to do with that process. The quests, the foraging for equipment and ingredients, the talking to the same people and repeating the same dialogue that I just did a couple weeks ago. The monotonous tasks that aren't relevant to my character's abilities.
That I fully understand and sympathize with - which is why I'm not trying to do everything in one or even two playthroughs. My current character is a Viking-ish Nord Warrior, heavy armor, one-handed, block and smithing. My next character will be some sort of warrior-necromancer (Death Knight, I believe some have called it) that uses one-handed, heavy armor, necromancy, destruction (fire) and smithing+enchanting...and wearing Daedric armor as soon as possible so he looks like Sauron's nephew who joined the family business. But regardless of what the character is, he'll be doing completely different quests.
My current character is doing the Companions quests, and some assorted others like Markarth (Forsworn) and this and that, and of course the Main Quest. Not sure what else, but no Dark Brotherhood, Thieves' Guild, Daedric quests or Civil War. The next character will be doing the Daedric quests, anything related to Necromancy, possibly the Dark Brotherhood, who knows, but there will be little overlap.