"Detract" means "to subtract value away from something". It can be applied to a vast variety of situations. For example, you could say "adding sugar to that dish detracts from its flavor". Or, "the high interest rate on that credit card detracts from its usefulness".
"Distract" is more specific. It applies only to ATTENTION. It has nothing to do with value. It only speaks of the taking away of attention from one thing and moving it to another.
For example, "Those hula dancers distracted me while I was trying to read a serious book". That means the hula dancers stole my attention away from the book I was trying to read. The hula dancers did not "detract" from the book (its value did not change), and they did not "detract" from me (because the value of me didn't change), though they certainly did detract from my EXPERIENCE of trying to read the book (because the experience became worse).
The hula dancers ruined my vacation which I had planned to spend reading. They detracted from my vacation!