It should be possible if re-writing the game's shaders isn't too difficult.
Shore foam, fading in a foamy looking texture the closer land or something gets to the water line, should be doable (though it's hard to make it look good).
I've that in the past on a DX9 game, it's doable:
[img]http://pix.wefrag.com/i/b/d/e/c/e/41609eea88428e50f2f7fa96ee127f81.jpg[/img]
Doesn't add that much complexity to the shader, eventhough I don't know how it might render on skyrim (high distance, lots of particles).
We'd need to unload these fxp archives to see what's inside the beast

.