I believe it gets reduced by magic resist first, and then the reduced damage gets reduced even further by elemental damage.
Like you get hit by 100 pts fire damage, with 50 magic resist, and 50 fire.
Magic resist reduces it to 50 damage, then fire resist reduces that 50 to 25.
At least that's how I think it works, I could be wrong, believe I read it on these forums.
not 100% sure about this as this is based interiely on subjective observations of my characters BUT, I think mitigation of the same type are incremental as in 50% magic resistance on a pot and 50% from armor enchant = 100% total (80% or so cuz of caps)
but nord frost resistance 50% + 50% magic resistance is multiplicated instead, as in calculated twice, an attack of 100 cold dmg get first 50% nord racial mitigation to 50 cold gmg, and then again a 50% from magic resist to 25 cold damage (not sure about this as it can simply be the cap and there are no logs)