For me, it's because once I've saved the world from a giant, planet-devouring dragon God everything else seems a bit... small by comparison.
This, "oh, so you're the guy that saved the world? Ok, you're now a footpad in the thieves guild, welcome, go race this beggar for a book."
But in Morrowind it made sense to be side tracked,infact you had to, and i looked forward to beeing able to continue the MQ.
In Oblivion, well... felt like i had to focus on it, and not to mention i didn't like it at all, i played that game every now and again for like 5 years, i've completed the MQ once, and that was last year...