Demiplane of Nightmares
The Demiplane of Nightmares is a minor plane within the Multiverse of Dungeons & Dragons, but one with a long history across different iterations of reality. As such, just what it is depends a lot on where you look.
Dimension of Nightmares
The "Realm of Nightmares" concept was first visited in the Mystara setting, which did not use the Great Wheel but instead used a more Sword & Sorcery by way of Michael Moorcock system of alternate realities, or "dimensions". One of the first of those dimensions to be detailed was the Dimension of Nightmares (or, simply "the Nightmare Dimension"); details were initially sparse, and the ultimate explanation was almost brain-bendingly confusing, but the basic idea is that the Dimension of Nightmares is a kind of prototype to the Far Realm we see in the modern era - a place "askew" from human perspectives, inhabited by alien, horrible and monstrous entities.
It was first mentioned in the module X5, and then again in the Companion Set for BECMI. Obliquely refered to in the module CM5, it would finally be spelled out just what the Dimension of Nightmares is as part of the Immortals Set; a parallel reality which exists at a perpendicular angle to the baseline reality, and which thusly is alien and weird by mortal semblances. This set took the first step towards asserting that, despite its name and the horrible monsters associated with it that we'd seen, like the demonic Malfera and the brain-harvesting Neh-thalggu, it was not actually a hellish netherrealm - it even invented a "human analogue" race native to the dimension, in the form of the Diaboli.
The Rules Cyclopedia and Wrath of the Immortals would retcon all of this away, stating simply that "dimensions" are entirely seperate multiverses from each other, and leave the Dimension of Nightmares basically its latter descendant; the Demiplane of Nightmares.
Other references to the Dimension of Nightmares include the Mystara Monstrous Compendium Appendix, the Known World Gazetter #3: The Principalities of Glantri, and its AD&D "reprint", Glantri: Kingdom of Magic.
Demiplane of Nightmares
The Demiplane of Nightmares was first mentioned by that name in Advanced Dungeons & Dragons 2nd edition, in the Planescape splatbook "A Guide to the Ethereal Plane. Here, it is simply mentioned as being a demiplane where nightmares are made real, with an obviously implied connection to the Plane of Dreams that it doesn't bother to actually fucking well elaborate on. No, we're not kidding: this is the entirety of its entry from that source:
Demiplane of Nightmares: Here's a place where a body's darkest thoughts and fears become reality, like living nightmares. The demiplane itself is terrible to look at, designed with strange angles and stranger geometry, and even a well-traveled blood's sure to think twice about staying in this realm for long. The creatures inhabiting the demiplane are horrific beings, monsters whose essence repels the most twisted bashers. 'Course, some cutters say that these nightmare beasts find other planewalkers equally horrifying, but no one's stuck around on this demiplane long enough to find out.
Finally, in Dragon Magazine #327, the article "Winning Races: Diaboli", which presented a 3.5 PC writeup up the Diabolus, that long-time inhabitant of the Dimension of Nightmares, what remains the definitive iteration of the demiplane so far... if only because no greater amount of information has ever been shed on it!
According to this source, the Demiplane of Nightmares is a unique demiplane that is coterminous with both the Ethereal Plane and the Plane of Dreams. Ordinarily, it would simply be a rather Limbo-like place of unfettered potential and dream-logic; chaotic, and perhaps scary, but harmless. However, it has also become partially coterminous with the Far Realm, and its maddening influence has begun seeping into the Demiplane of Nightmares, turning it into a hellish place.
The last reference to the Demiplane of Nightmares since then is in the Creature Catalog of Dragon Magazine #343, where the writeup of the malfera states that the Demiplane of Nightmares is the demiplane where "evil dreams are grown and planted within mortal minds".
Due to its limited presence in literature, we know very little about what inhabits the plane. Mystara lore establishes the Neh-thalggu, the Malfera and the Diabolus as native to this plane. The Winning Races; Diaboli article also mentions the existence of feyrs, nagpas and, for some reason, maelephants - likely meant to be malfera, but misnamed as a result of the author confusing the two different species of fiendish elephant-headed monsters of D&D's past.