But them not drinking water IS realistic for the time. Beer was drunk in those times instead of water due to the water being so 'dirty', and there where no filtering or anything, so putting water as something to drink would actually make it UN-realistic.

Depend, the water look good to me, no I would not drink anything downstream from an city or near the hot springs. Would also be careful about standing water especially in dungeons.
Little water pollutions as the population is low.
Might say that the lack of drinkable water is of the same reason as it's a lack of bottled air. You don't care about air unless you are diving or in space. You care about water in the desert not where it's everywhere.