Phones do not have "hooks" anymore. In the old days, when a phone was not being used, its receiver (the part that goes against your face) would hang on a metal bracket. The weight of the receiver would pull the bracket down and hang up the phone.
So, a phone that's "ringing off the hook" means it keeps ringing so much or so often, that it falls off the hook.