Kai, the first one is correct- when you say "i read that book of yours", you are implying that they have owned the book and possibly you are borrowing it, or that they are the author of the book.
For the second to be true, you would say, "i have read that book about you." meaning, the book was about the person you are speaking to.