It might help you to look at the little words - on and down - if you want to understand the difference between these phrases.
'Off' is the opposite of 'on'.
So if the book is on the table, it can fall off.
The leaves are on the trees, and then when autumn comes they fall off the trees.
'Down' is the opposite of 'up'.
You build a house made out of cards (up..up..up) but then eventually this house will fall down.
It's really quite simple if you think about what the words mean. These aren't phrasal verbs or idioms, they are just little words that mean what they on/off and up/down.