"It was a monster mash"
- – Bobby Pickett
Monsters are creatures (usually assumed to be non-human) that are some combination of hideous, hostile, and dangerous. They came into being in the misty distance past as mankind at the time, being a bunch of cave dwellers with rocks on the end of sticks for weapons, started to imagine all the possible horrible creatures out there, bigger, nastier and more mysterious than any animal they knew. So was born monsters, reflecting all of those terrors that mankind believed might come after them.
Historically in traditional games, monsters have been used as adversaries for adventurers and parties -- as they are not human or human-like, and are always Chaotic Evil (or, if they are not intelligent enough to have an alignment, they are ravening, dangerous beasts), they can be killed for XP and loot without causing any guilt for the players later. They are frequently listed in books dedicated to just monsters, the so-called Monster Manuals.
This has changed in more recent years. Games nowadays tend to take a more nuanced approach -- some species are described as tending towards evil, or as being so dangerous that they cannot coexist with humanity (or elf-kind, or dwarf-kind, or whatever the local intelligent player character species options are), but seldom are PCs given license to slaughter anymore. For example, the old standby monster races, like kobolds, orcs, and drow, are much more fleshed out in more recent role-playing games; some members of these species are friendly (if frequently misunderstood), or even playable as characters, and even the "evil" ones are given detailed societies and complex motivations. For games set in modern or futuristic times, mutants may still be dangerous and needing to be put down, but they are often cast as victims of circumstance. Conversely, many games are coming to include certain humans in the list of "monsters", particularly those who commit terrible crimes -- for example, the "Slashers" from the World of Darkness's Hunter: The Vigil line.
A possibly related phenomenon is that of the so-called "monstergirl", where monsters are anthropomorphized, and in particular given physical characteristics of beautiful human women and a human-scale intelligence, if they didn't have one already. Note that this does not necessarily result in a less dangerous creature (some people enjoy that sort of thing, after all).
Of course, some monsters resist all attempts to make them sympathetic, and will provide guiltless XP for generations of gamers to come. Zombies, skeletons, and other unintelligent undead creatures are probably the most common variety of such monsters for fantasy games, while games set in the near-to-distant future often use preprogrammed robots for the same purpose.