Okay so I married Sorex Vinius from the Winking Skeever in Solitude, and at the time I had the Breezehome house in Whiterun, but now I have a house in Solitude, and Riften. I can't seem to find him anywhere, not even in the Winking Skeever. So where did he go?

Most likely he's still at the Temple of Mara in Riften.