I can't argue there. I started reading the first novel of the series until I found out there were Elder Scrolls novels. For some reason I never went back to the Game of Thrones book.
Your loss, because the Song of Ice and Fire series is one of the best series of novels I've ever read.
To answer the OP yes I can see some connections to ASoIaF.
1. Biggest connection, dragons have returned. In ASoIaF they are little, and there are only 3. Even in A Dance with Dragons they are a lot larger but still small, not yet the epic size of a dragon.
2. The king is dead, and civil war has spread throughout the realm. Now the war in Skyrim isn't the war of 5 kings like in ASoIaF, but it's still similar. The death of the king, and Eddard, and the rise of a boy king who's legitimacy is called into question leads to civil war in ASoIaF.
3. Dragon Born reminds me of the Targaryen's who have within them the blood of dragons. There are no shouts in ASoIaF though, but there is a language dragons can be taught things in. Daenarys demonstrates this in A Storm of Swords. Dracarys!
4. The overall setting of Skyrim brings to mind the realm of Westeros. This could be because I've recently read all the books in the series, and I maybe superimposing what I imagine to be Westeros on to Skyrim.
5. Drauger's though they appear in Norse mythology which is what the Nords/Skyrim are based on, they remind me of the White Walkers, more specifically the Wights that the White Walkers turn people into. Blue eyes, pale skin, walking dead.
6. You could say the Dark Brotherhood are kind of similar to The Faceless Men in ASoIaF, since they are basically assassins, and give people the gift of death.
7. This isn't too big of a connection, but giants are also in ASoIaF, and they ride mammoths. The giants in ASoIaF are a lot more hairy than the giants in Skyrim.
There is one that involves a similarity with Ulfric, but I won't mention it because I don't know how to use spoiler tags on here, and don't want my post deleted.
I'm sure I could come up with more, if I thought about it for a while.