System Internals (Free) from microsoft comes with a program called Process Explorer. (google that) It will show you that YES you can indeed go over the Actual Vram your card has. I have been 300 MB over the 1024 my card has. Todays modern Video cards will ask the System for its Ram. This will cause you to have sever stutters and can cause you to go from 50-60 FPS to 5-10 FPS.
Dont trust what i tell you , download "Process Explorer" it will show you what none of the other utilities will . It can actualy Replace "Task Manager". The following is a screen shot of Skyrim running for 19 hours. the shot after is showing that my system was using 250 meg of System RAM.
http://i232.photobucket.com/albums/ee72/camaro_69_327/Skyrim/Skyrim19_hours32Mins.png In the top header of that screen you see the name of the program and the Address to get it.
http://i232.photobucket.com/albums/ee72/camaro_69_327/Skyrim/TESV_2012_01_01_11_38_01_055.jpg Looking at the bottom graph is System mem at 250 MB I was getting stutters , it takes 3 hours to fill that graph all the way across like that ..

Now even tho the Vram was not filled to the top, it starts using system ram when the card sees that what is being asked for would saturate the Card. So instead of using the last of the available Vram it just uses your system ram. I used this to tune my Game to use only 900 MB so it never goes into using System RAM.
You will not see this using Gpuz or MSI or what ever else you want to try. Least i never found one showing me what my GPU was really using. This does ..as you can see. The top Graph is the GPU (engine) usage. the Middle graph is the VRAM (GPU) bottom is RAM (System).
You can get Process explorer on its own or get the Suite. It plays very nice with Skyrim as you see i ran them together for 19 hours straight! Your mileage will vary.

Till you see what your System is giving your GPU you can be Overfilling it and never know! I didn't.