Swimming in various parts of the Vale cave system, I get pulled straight to the bottom. Luckily I'm an Argonian, so it was just a mild annoyance. Swim closer to the sides, it happens less that way.
I'm almost positive this is it. It's a bug, but a very, very minor one in comparision. It occurs in many different bodies of water. It's kinda just an accident where they didn't get the water to properly cover the entire area it's supposed to.