He has written a book whose name I have forgotten.
Only people can be described by who, whose, or whom while things are described by which, what, and that
He has written a book the name of which I have forgotten.
You would need to add a comma before “the name” because it’s an appositive phrase. Otherwise this one is technically correct, but putting “of” in front of “which” is so “grammatically correct” that we associate it with old books and stuffy formal writing.
He has written a book that I have forgotten the name of.
This would be a good popular form. Although it’s weird to put of and the end of the sentence for on-native English speakers, you kind of just have to think about the rebel nature of American pop culture and speech, and kind of switch the order up and make some mistakes in your english on purpose. Just look up which ones everybody makes beforehand. (:
He has written a book of which I have forgotten the name.
This one is also correct, but it’s again in that formal tone (equivalent to the “name of which” one.
Keep up the good work! And just remember, when it comes to English, the rules barely make sense even to the kids in American schools, so straight memorization is key to most English grammar.