A Splatbook is a non-core sourcebook for an RPG that provides additional rules and material that can be used with the main system. An example of a well-known splatbook is the Book of Weeaboo Fightan Magic.
The White Wolf community is often cited as the origin of the term. The names of the supplemental material books for World of Darkness would vary depending on which branch of the game they were for (clanbooks, tribebooks --say that three times fast!-- , tradition books, etc.), but they still all followed the same naming conventions of [SOMETHING]books. On Usenet newsgroups, these came to be known as "*books", after the asterisk's function as a wildcard character in computing. Common CIS slang calls these "splats", because they (kind of, sort of, if you squint) look like star-shaped splatters of paint, slime, blood, or some other slightly viscous fluid.
Splatbooks tend to be known for contributing to Power creep. In order to make the new books more popular and desirable, they often contain significantly more powerful classes, feats, etc. than the core books for the respective system. Since later splatbooks would need to 'compete' not only with the existing core rules, but other splatbooks, later releases may include even more powerful rules. Players obsessed with splatbooks are usually either interested in the mechanical advantages, or in playing a special snowflake character that isn't available in the core book.