Installing 8gb seemed to work fine.... 12gb or more and things got problematic.... worse as you further increased it.
Doubt that's the issue. I have 16GB of 1600mhz RAM and I play with ugrids = 7 without any issues at all.
Perhaps it's VRAM? I run with a 2GB GTX 560 Ti. Two of them in SLI as well, which although doesn't double VRAM to 4GB, might improve stability in other ways. This game hogs VRAM though, so perhaps 1GB isn't enough to run a high ugrids value with decent stability?
EDIT: My ini values, if you're curious:
uExterior Cell Buffer=64
uGridsToLoad=7
iPreloadSizeLimit=51380224
[Papyrus]
iMinMemoryPageSize=100000
iMaxMemoryPageSize=5000000
iMaxAllocatedMemoryBytes=1800000000