Imagine you've just eaten a flaky croissant for breakfast you are covered in pastry crumbs. Before you go to work, you 'brush yourself off' - brush away the crumbs so that you look presentable. You leave the crumbs on the floor of the cafe, walk away, and then get on with your day. This is a literal meaning of the phrase.
When someone is unhappy about something that's happened, we tell them to do this metaphorically. Wipe away the tears, straighten your tie (if you're a man), re-apply your lipstick (if you're a woman), and walk away, leaving the metaphorical crumbs behind you. Forget about the bad experience, and get on with your life.