I don't. Between characters I mean.

The thing with any game like this is it inevitably becomes more and more familiar. Even if you really take your time exploring the map, you can't just unsee what you've seen. I'm finding with Skyrim I'm semi-consciously trying not to keep track of which dungeon is which and certainly not paying attention to which word wall has what word on it. I'm trying to avoid starting a new character and thinking 'I like this shout - so this word is here, the second is over there...". Skyrim is so big and has so many dungeons it's a bit easier to start getting a bit mixed up.
I was just in a dungeon and thinking this as I know I've been there before but can't quite remember what the layout is, what the boss is and what wordwall it is (I remember it has one). I class that as a victory over my generally pretty good memory. Like when you remember where all the enemies are positioned, you lose the surprise factor, make less mistakes and it gets easy. It's a kind of willful amnesia in an attempt to keep the game fresh for as long as possible before I know it all as well as I know bleak falls barrow.

Does anyone else relate to this or am I a bit mad?

About 10 years ago, I decided that my memory is going to be replaced by the Internet. Since then, I keep track of things for games through Wiki's.

I keep to one character at a time so as not to have to tell myself not to... um, know things before I see them. Unfortunately I do tend to remember things in video games but nowhere else! Seriously, my memory is so bad otherwise I'm surprised I even remember how to put on underwear.

In answer to your question, I don't think there's anything mad about it.

I also tend to make an attempt to not memorize things. Like Lynzi, I seem to only have a capability to remember things within video games, whereas in my life I tend to forget just about everything, especially important names..

But, in games like Skyrim, I know I will someday be playing another character through this same landscape so I endeavor to not remember where things are and how things are located. But I know that when I do eventually play through this all again I am going to go about it differently. Since I'm on my first playthrough still there is tendency to see everything and go through every pocket and chest and drawer. But when I play a mage that will not be the case. As a thief it makes sense. But my mage would not care about such tedious things.