if anything, stories and legends always play up the villain, if they had this power it would be known (ebsdies, they are clearly not trying to hide, so they would be known anyway)
An intelligent villain spreads lies and misinformation.
having a fortress in the first place does nothing for the whole "stealthy" thing
A fortress on an island within a sea of ice shrouded in mist. Ships are unlikely to be able to sail there. Normal peopel aren't going to take a dip in those icy waters. This makes it a very secret place.