"Don't screw it up" basically means don't mess up/don't destroy it. It also depends on how you use it in a sentence.
example: "I'm counting on you to drive your sister to the mall today. Your father just washed the car. Don't screw it up."
OR
"This is a painting my mother did when she was a child. It is very special to me. Hang it up and DON'T screw it up"
Alternatively, you can say just "don't screw up."
example: "You only have a month left until summer vacation. Don't screw up and start missing a lot of days, or you'll be in summer school."
OR
"Don't get nervous on your road test and screw up."
I hope that helped.