That's how I see it as well... His temple, thanks for the correction, is basically a 'time line' of his transition from Dragonpriest to Herma Mora follower.
Which basically goes to show that the confrontation between Vahlok and him was before his transition.
I'll try to make an accurate timeline, based on what we've discussed here.
Miraak's Timeline:
Dragon Priest -> Turns on Dragons and is defeated by Vahlok, causing him to be locked inside of Vahlok's tomb with him -> Vahlok dies/is murdered by Miraak -> Miraak begins to build his temple nearby Vahlok's tomb and his former jail -> As Miraak does so, he begins to become consumed by worship of Herma and gaining power and knowledge -> Is eventually found/attacked by the dragons, who overwhelmed him with sheer numbers and defeated him -> Is taken to Apocrypha by Herma-Mora before his death -> Thousands of years doing whatever he does -> Events of Dragonborn DLC.
This is the best I can do. Note, to everyone reading this, this timeline is based on speculation and reasoning and conclusions drawn based on in game lore, not solid fact confirmed by any source, in game or otherwise.