Due to an armor mod being released that creates Gregor Clegane's armor from Game of Thrones, I want to switch from wearing light armor to heavy.

I like my character and don't feel like starting over again.

Any major downfalls to switching from light to heavy armor at level 32 other than the reduced armor rating?

wasted perks?

That's the only real loss. You can skill up to overcome the reduced armor rating.

And being PC.... you can just console command to move perk slots if you really want.

You will "waste" some perks, but there is nothing major outside that other than that.