Refers to author Jack Vance, perpetrator of the 'Dying Earth' fantasy fiction series of which Gary Gygax was a big fan. In roleplaying games the term 'Vancian' became notable because the Dungeons & Dragons magic system, specificaly the way wizards use spellbooks and memorize a certain number of spells per day was based on the magic system Vance used in his books.
Also called "fire-and-forget" spells because after you've cast a spell, you're unable to do it again until you get a good night's sleep and at least an hour to re-study your spellbooks. In Dungeons and Dragons at least, in addition to restudying the spell books, the wizard is also casting most of the spell, and the casting time in the spells stats is just how long it takes him or her to finish casting the spell. Systems that use Vancian magic will also have artifacts with the same limitations, such as scrolls that can be read only once to cast a spell written on it, or wands that use up a number of charges as magical ammunition to cast spells.
4th edition has dropped the spellbook memorization mechanic from previous editions but retained similarity to Vance's magic in the distinction between simpler at-will spells, one-time spells and complex rituals. 4E casters never run out of spells as older edition wizards did (due to memorization issues), but rely on at-will spells (can be cast continuously, without limit) and magic implements (ie wands) for a more Harry Potter feel.
5th edition has introduced a hybrid of both systems. All spellcasters now have independent spell slots (representing their power reserve) and an amount of individual spells available to them (either picked from a larger list for the day (wizards & clerics), or locked in at level up (warlocks)). They can then opt to use a slot to cast any spell available to them at the moment, with the simple caveat that the more powerful the spell, the higher of a slot it requires. At-will spells (called cantrips) still exist as well: while weaker than 'regular' spells, they're always available and require no slot to use; meaning the spellcaster can always do something, even if circumstances forced him to spend all of their slots.