Calixto's house seems to be permanently master locked. I can pick it, but then he tells me to leave. This seems to happen 24/7. Is this another bug with the already horribly bugged Blood on the Ice quest?

It's hard to say considering you did not share with us whether you even have that quest, what stage you are on etc... I did Blood on the Ice and didn't have any issues, on several charachters. I know that quest is notorious for bugging out though. But we need more info from you I think.

I have the quest. I retrieved the two journals and the strange amulet from the house. At this point I talked to the old woman and told her about the journals. Now I am trying to get Calixto to talk to me so I can give him the amulet.

Likewise I also thought that at one point the quest did bug out it just took some detective work to figure out what to do next.

So it's not a bug that Calixto is locked in his house 24/7 and will not talk to me at all?

He does seem to lock himself in for long hours at a time, if waiting doesn't make him open up I would suggest leaving the city for a while and returning a bit later, this may "reset" his shop and cause it to be open. Of course I recommend making a save beforehand because we all know that quest is a bit dodgy.

According to Wiki, he is awake from 5 am til midday is his shop open during those times?

I saw that on the Wiki as well but no his shop is not open at any time.

Well, make a back up save and fast travel to a few places then return and time it so you enter Windhelm between those times (9 am?), hopefully that will cause the shop to be open. It's worth a try, nothing to lose there.

try talking to that other guy in the palace (starts with a j?) first.

If you show him the amulet without accusing the court wizard, he appears to lock himself in until the quest advances. Either talking to the wizard or accusing him should advance it, however, showing/selling Calixto the amulet seems to turn him into a recluse until you get to the next possible murder.

Completing the quest in this way, selling the amulet to Calixto then talking to the wizard instead of accusing him, can also cause Calixto to attempt his next murder in broad daylight. I'm unsure if this is just a bug with the quest, intended, or due to the game un-syncing from the clock (which is really bad in my current playthrough).

I ended up just going to wizard and forgetting about the necklace. I then went outside and Calixto was trying to murder someone in broad daylight.