Both sentences have a similar meaning, but with some small linguistic difference.
"Sobre", is a preposition and when it's used in the sense of your sentence, means that "the book has a support point on the table", it's resting on the table. But remember that you must say "Hay un libro sobre la mesa" or "Hay un libro que está sobre la mesa".
"Encima", is an adverb wich indicates in this sentence 'a place more or less immediately superior', but not necessarily in direct contact with the table. The book could be placed on something else that is also on the table. "El libro está encima de la mesa, sobre mis papeles".
Anyway, in Spanish is perfectly understandable in either of both cases, and are used interchangeably in everyday language.