This makes sense. That would be akin to losing the skill points that were needed to get a perk to get rid of the perk, and it would be appropriate to have to retrain the skill, and relearn the perk. What's not appropriate is to have to redo EVERYTHING I've ever done on the character. Why should I have to redo all the quests I've done, retrain ALL my skills instead of just the relevant ones, and reacquire all of my equipment? None of these things are relevant to the acquiring of skills. By following the way learning works in the real world, we forget less used knowledge and replace it with that which we want to learn and employ. There should be a similar system in game if we're talking about realism.
I could also get behind the idea of using dragon souls to fuel new perks, provided it was at a fairly steep rate, and maybe a vendor as a last resort. And a quest or series of quests to respec or retrain or whatever would be awesome.

Can you not see that your argument essentially tries to justify any and all absurdities, no matter how ridiculous, just because it's a single-player game?