Hi Haru,
FORMAL vs INFORMAL is often a matter of vocabulary, 'rarety' and sentence structure.
To be more specific and in reference to your examples:
1. 'this kind of book' is more commonly used 'thus' 'a book of this kind' could sound more formal (unless you are pointing at it:-)...
2. the same goes for 'these kinds of books'
As for 'a book of these kinds' as unusual (and out of context) as it may sound, it could have a meaning of sorts in a sentence such as:
3. 'I read a book of these kinds of short stories recently...' It does not sound formal (even limit English) but could be conversational like ^_^.
Bye bye, for now