Please, please, please don't take what I'm about to say personally. I think your question is one that a lot of people have probably wondered about at one point or another. I even up-voted the question. Unfortunately, the answer is "it doesn't work that way". And there is no good way to answer your question without possibly making you feel bad for asking it-- which is a shame because I think that there is useful info in the answer. So, bear with me on this and understand that my motivation is to explain how this stuff works and not to make you feel bad.
Modern displays (TV's, LCD's, Plasma, most CRT's) are raster based devices. Meaning, they redraw the screen one scan line at a time. The video interfaces used to talk to these displays are based around a raster interface. NTSC, VGA, HDMI, etc. are all raster display technologies.
Years ago, mostly in the 70's and early 80's, there were some true vector displays. Instead of drawing things a scan line at a time they actually traced out the shape of the objects. The best examples of this are analog o-scopes, and the arcade games Asteroids and Battlezone. There are very few color vector displays that were used, and the best example of that is the arcade game Tempest. A different form of vector display is the systems used for laser light shows.
The electrical interface to a vector display has signals for the X and Y location of the beam, the intensity, and sometimes the color. This is very different than the interface for a raster display.
Older CRT technology could be used for either vector or raster displays, but LCD's, Plasma screens, OLED's, etc are all fundamentally raster based and cannot easily be used in a vector type mode.
The problem with your question is that you are asking about a "vector based VGA graphics card". "Vector Based" does not go with VGA. VGA is a raster based display standard and interface, while vector is not. You cannot mix the two, since they are different systems.
It is possible to make a vector graphics card based on an FPGA, but you can't connect it to a VGA type monitor without doing a vector to raster conversion in the FPGA-- and that requires as much RAM or more than a standard raster graphics card. You could get a vector display, but those have all but gone away with the demise of the CRT.
The easiest way to get a vector display these days is to get an o-scope that has an X/Y mode. In fact, there are many projects on the web that use an old o-scope with CRT tube as the display for something. Here is a project that uses one to make a clock. And here is another one. There are dozens of other similar projects on the web.
While interesting, these projects are little more than novelties. They are cool novelties, but just novelties. And none of them approach the display quality of a normal VGA card.
An alternative solution is to simply get an FPGA with some external RAM. There are many ways to do this, but the easiest and most pain-free way is to use a Xilinx Spartan-6 based board with external DDR2 SDRAM. There are several FPGA development boards on the market that have both of these chips that will work. Some of them even have VGA interfaces too.
I would not use external SRAM. SRAM, especially async-SRAM, is going to be slow to interface too and the memory size will be limited. It's not impossible to use SRAM, but it is not any easier to use SRAM than DDR2 SDRAM on a Spartan-6.
I would also not use a Spartan-3. The S3's DDR SDRAM interface is not very good, and is difficult to get the signal timing to work out reliably. The Spartan-6 has a "hard macro" for the DDR2 interface, which makes the whole thing much easier. The S6 also has more internal RAM for buffers, FIFO's, etc. Xilinx has a nice "Memory Interface Generator" core that makes interfacing several different chunks of logic to the DDR2 SDRAM much easier (with multiple read/write ports too).
As for an open source Vector Graphics core-- I do not know of one. That doesn't mean there isn't one, just that I have not seen one. But I also doubt that you'll see one that is very good. There just isn't that much call for one since the displays are somewhat rare and limited. If you do find one, it will likely be fairly specialized (it only displays a clock).
Get an FPGA board with a Spartan-6, DDR2 SDRAM, and a VGA port and you'll be much happier.