From 1d4chan

Mentalist is another term for individuals who possess the mind-related powers of psionics, with something of a focus on telepathy and clairsentience. In Dungeons & Dragons, the Mentalist is a "School of Effect" Specialist Wizard, which debuted in the Advanced Dungeons & Dragons splatbook "Player's Option: Spells & Magic" alongside its fellows the Elementalist, Dimensionalist, Force Mage and Shadow Mage.

In a nutshell, mentalists are described as somewhere between an arcane emulation of psionics and a variant Enchanter, one specialized in spells that directly affect, control and otherwise manipulate the mind, as opposed to the Enchanter's broader spectrum of empathic and telepathic trickery - as such, they cross over with the Illusionist to some extent as well. The mentalist is a wizard who deals in subtleties and influences. The mentalist is not at his best in open battle against hordes of enemies, but he can be an extraordinarily effective character in investigations or confrontations against single enemies.

Mechanically, it has the following traits:

Ability Score Requirements: Intelligence 15+, Wisdom 16+
Prohibited Schools: Transmutation and Necromancy
Can memorize 1 bonus spell per spell level each day, which must be a Mentalist spell.
Creatures suffer a -1 penalty to their saving throws against Mentalist spells cast by a Mentalist.
Mentalists gain a +1 bonus to saving throws against Mentalist spells. This increases to +2 at 8th level.
+15% bonus to attempts to learn Mentalist spells.
-15% penalty to attempts to learn non-Mentalist spells.
Cannot learn spells from Prohibited Schools.
Automatically gains one Mentalist spell in their spellbook each time they reach a new level.
When attempting to research new Mentalist spells, the spell's level is reduced by 1 to determine how difficult it is to research.
At 11th level, can Detect Charm or Mental Influence 3/day by pointing at a desired target and concentrating for 1 round; this only targets one individual per use, but otherwise mimics the priest spell Detect Charm.
At 14th level, can Dispel Charm or Mental Influence 1/day with a 50% chance of success, modified by 5% per level/Hit Die difference between the mentalist and the caster of the charm. The mentalist must be within 10 yards of the subject and must concentrate for one round in order to use this power.

Despite their name, Mentalists do not appear in Pathfinder, although they share similar conceptual ground with the Mesmerist.