that's roughly my specs. What I did was run it in high textures, but at 800x600 instead of 1280x1024. Things look slightly fuzzy, but it looks better (to me) than 1280x1024 with medium/low text.
Point is, you'll have to fiddle a bit to get the best for your system, but it's no biggie really.
Hm...I have 9650m GT 1 GB in my laptop, witch is a good mobile GPU but is still nowhere to desktop 9800 GT in terms of performance...and I could play FO3 at high settings with native resolution....so I can't belive that you were forced to play in 800x600.....even tho' you have only 2 GB of ram...it's just impossible that I run it better just because I have 3 GB of ram, you have powerful GPU so there must be something bad with your sysem, if FO3 lagged on your card....