Won't comment on what I think of the idea in general, but....
I'll make notes and draw maps in a journal to keep track of places I've been.
...you do realize that Skyrim isn't entirely a caveman-level uncivilized wasteland, right? Even if you decide to do this hand-drawn-map-only thing at the beginning, once you reach a town of any decent size (above village... anywhere with a bookshop/etc), you could buy a paper map of the province.
Oblivion (with both the Black Horse Courier, and the amazing number of books people had on shelves) taught us that printing, as a craft, is well developed in Tamriel. It's not monks carefully hand-writing and illuminating two manuscripts a year, for only the ultra-rich to afford a couple volumes. They were handing out newspapers in the streets - paper, ink, and printing are relatively cheap and widespread. There is no reason, none whatsoever, for your "realism"

to include the inability to use a map.
(Now, the in-game 3D map, on the other hand....

)