The expression is an idiom, and idioms often cannot be understood literally (word for word). The picture that the expression paints / the message in the exprssion is that we feel a little bad if it rains, so the problems that we have in our lives are like the rain. People feel blue when it rains and they feel blue (sad/depressed/unhappy) when they have problems. But if there is too much rain or if we have too many problems it feels like the worst day of our lives.
Now I will tell you a story:
One morning it was raining hard and that made me sad because I had planned to have a picnic at noon. Then I decided to make some breakfast, but I burnt the food. I was hungry, so I ate the burnt food, and the hard, burnt bacon cut my the inside of my throat. That made me angry so I took the plates to the sink and the plates slipped out of my hands and broke on the floor. When I tried to sweep up the broken glass and porcelin I had to bend down and I felt a pain in my back. The I laid down on the couch to rest my back. I had a cigarette and fell asleep. The house caught on fire, but I woke up in time and escaped from the house. My car was parked in the garage attached to my house, so I drove it out onto the street to prevent it from burning. Unfortunately, as I was backing out of the driveway I was watching the house burn down and I failed to see the truck that was driving down my steet. Naturally the truck hit my car. I wasn't hurt, but the car was completely destroyed......
It didn't rain that day - it poured.
Everything that could go wrong, did go wrong.
A pouring rain is the same as a torrential downpour, like standing under a waterfall. When our life is so full of so many problems / so much rain we become incapacitated, unable to do anything, overwhelmed and rendered useless.
On that day, when it poured, I was powerless.