The only possible reasons for this (to be honest considering the engine, it could be anything), is that the zone changes or things that need time calculations like fast travel, carriage ride and wait, is doing it wrongly, almost as if there was a margin of error in place (a guess).
Seems to be my experience as well. I started my second character with the intention of not using fast travel. The game clock starts off 1 hour out of phase, but was easily corrected. The game clock stayed in perfect working order up until the first time I got lazy. I didn't feel much like walking back DOWN the 7000 steps. Sure enough, the problems began immediately. I fixed it. Things stayed together until the next time I "fast traveled" by using the carriage. Blam, clock is out of sync again.
So I
think the main trigger is using fast travel or the carriages. It doesn't seem to get bothered at all by simply walking everywhere, but even I will admit walking everywhere gets tedious after awhile when quests want you to go to Riften and you're in Solitude at the time.
I'm still on the original 1.1 if that makes any difference. I'm seeing reports that Patch 1.2 or 1.3 may have fixed this? If so I'd certainly love to know since it would be compelling enough IMO to warrant allowing Steam to do its thing. Other than this time bug, the worst I've seen is a few broken quests - which I think got that way BECAUSE of the time bug. One CTD in collectively 190 hours of play across 2 characters.
EDIT: Well, ok. So 1.3 hasn't solved it yet? Consistently changing at noon instead of midnight?