Well, that's a big ole can o' worms.  Every book artist seems to have their own definition, but traditionally, the book arts fall into several categories.

Artists' Books: The Smithsonian defines them as "a medium of artistic expression that uses the form or function of “book” as inspiration. It is the artistic initiative seen in the illustration, choice of materials, creation process, layout and design that makes it an art object." They can be handmade or machine made, unique or created in multiples.

Democratic Multiples: Democratic multiples are inexpensive artists’ books sold cheaply or even given away to as many people as possible. The term was attributed to Clive Phillpot who headed the Franklin Furnace collection, a collection of artists books produced after 1960. The website says it will "accept any multiple published by an artist as a work of art." In a practical sense, though, it isn’t truly democratic. While the price might have been low, they had limited distribution. And as Johanna Drucker points out, there is no guarantee that the content is accessible to everyone (thus, not democratic). Today, with the possibilities of the internet and print-on-demand, democratic multiples are being reinvented. 

Fine Press Books: Fine letterpress printing made up the bulk of the earliest European-style book art: it employed fine papers, a good design, and excellent printing, not unlike a livre d'artiste, but the content was and is often generated by the maker or in collaboration. It can be combined with other terms, such as fine press livre d'artiste, or fine press artist's book.

Livre d'Artiste: Book designed to hold images, usually prints or illustrations, by a famous artist. Picasso and Matisse were especially prolific makers of livres d'artistes.

Sculptural Books: Sculptural books are works of art that begin or end with the book form. In the former, books are often manipulated by dis-assembly and re-assembly to create new structures. The popular terms "hacked books" or "altered books" can also sometimes be used to describe the resulting structures, although these terms are generally applied to more crafted objects. For the latter, various materials are reinterpreted to form book-like structures. Brian Dettmer is a well-known maker of sculptural books.

Wikipedia has a very good beginner's overview of the history of the artists' book with plenty of references.

Sources: http://makinghandmadebooks.blogspot.com/2011/04/defining-book-art-whats-in-name.html and http://thebookartblog.blogspot.com/2010/05/what-is-book-art.html

An exhibition catalog featuring approximately 120 works from Otis College of Art & Design Library’s Special Collection of 2,100 artists books dating from the 1960s to the present.