Examples of correct statements are: The book is difficult to read. Read the book in the library. This book is to be read in the library. (The last two have similar ideas) We are not to be blamed. Don’t blame us. (Similar ideas)

Your first sentence feels wrong to me because the actual subject is "to be read", not "book": "For this book, to be read is difficult." For the second sentence, both ways are good, but the meanings are different. When you say "We are not to blame" "To blame" is an adjective clause and the sentence means "we are blameless". When you say "We are not to be blamed", "to be blamed" is an adverbial clause that modifies the verb of the sentence, "are", and the sentence means "one should not blame us". I use "one" because the passive nature of the sentence does not specify what the subject should be.

The natural way to write or speak is to use the infinitive : "The book is difficult to read" "We are not to blame" Using the passive form 'be' + past participle does sound unnatural in the wrong context, but would be natural in the correct context. It is usually applied in situations describing the possibility of the act eg 'If that was the case [ie some condition applies] then they are to be blamed'. You can equally well use the infinitive form. Another situation in which the passive form is used is to enquire about doing something : 'Are these [some items of food on] to be eaten ?' There is a possibility that they are only for display purposes. A similar scenario : 'Are these [items] to be sold ?' [or are they just for display] If you stick to the infinitive form you will always be safe, as it is always correct and natural.

Both ways are correct, but it is clearer and more efficient and more natural sounding to omit the word "be" in both of these sentences.

They're different structures. Without 'be', the infinitives are behaving as adjectives. With 'be', it's passive voice. They won't necessarily have the same meaning. In the first sentence, the pronunciation - and the meaning - are different. The book is difficult to read [infinitive - reed]. - This means that people find it difficult to understand. The book is difficult to be read [past participle - red]. This doesn't really make sense - I can't think what it would mean. We are not to blame. - It wasn't our fault. We are not to be blamed. - Whether it was our fault or not, people should not blame us. So it's not a question of being natural or unnatural, right or wrong. They are different structures that usually have different meanings.