Anyone having good experiances with this? Or are baddies too op to do this on. I figure pairing this with deep block and it becomes viable.

Conjuration or a follower provide enough destraction that you can great critical charge to your heart's content

It wont be a problem.Heavy armour is not better than light in this game for mitigation and you will gain the benefits of faster stamina regen from the perk on the light armour tree.