"I refuse your order" would most likely mean that I will not agree to accept or receive your order from you. For example, you hand me an order form and I won't take it from you. Another example: I might refuse a package by telling the delivery person to take it away.
"I reject your order" could be used the same way, but the word reject suggests that I looked at your order, considered it, and decided to say no. Maybe I accepted your order form, but when I looked at the form, it had several errors, so I rejected it. I will not honor or agree to the request.