Hmmm nope, you'll get to a point where your textures will make no difference, you could have a 16k texture and it'd still be the same quality as a 2048x2048 texture. Just because the game stretches it does not mean it'll look better larger. Infact it may look worse as your details are stretched.
A 16kp^2 texture will almost always look better stretched than a 2kp^2 one, simply because it has more pixels so each is stretched less (less actual scaling). The only possible time it will look worse is in extreme compression, dropping pixels will make a very grainy texture. As long as you don't make the 16kp by stretching the 2kp, an actual 16k^2 source will look great. In fact, it's complete overkill.
The break-even point for quality here is when one screen pixel matches one source texture pixel. It's not particularly easy to line up, since it goes from source texture to UV map, model view and projection, to final screen resolution. More often than not, you can get away with 2-4 or even 6 source pixels per screen pixels. That's without considering tiling: texture size goes up exponentially as resolution goes linear.
FPS will certainly suffer as size increases, by a large amount. Textures start bouncing between the video card and RAM every frame, which gets incredibly slow (relative; you can only transfer a few dozen GB/s, so swapping 1GB of textures every frame at 30FPS gets expensive). There are documented stability issues with the Bethesda strain of Gamebryo and texture size, plus limitations of older hardware. In particular, D3D9 imposes limits which may be as low as 4096, although more often 8192 cubed. For now, the absolute upper limit should probably be around 8192x8192x8192x128bpp (not that any of the series supports volumes or HDR in full). For reference, that's a 192GB texture or an 8KM cube in Minecraft. Way more than you need unless you're megatexturing an entire landscape.
Long story short, you have to find a balance of source:screen pixels and size:resolution. Whatever gets enough pixels on-screen without being too big, and tiling and texture atlases were both designed to help with that.
The sweet spot for most hardware and quality these days is in the 512-2048 range, with smaller and larger for exceptional textures (very small items or very large ones that can't repeat). Normal and relief maps play into this by adding some detail, and proper use of tiling will make a texture go much further. 512 will often look blurry, much more than 2048 won't be noticeable unless it's really spread out. Normal maps can occasionally be half-size and reflection/environment can easily be half or quarter, which saves that. It really comes down to how much screen the object will take up, and how close the player will get.