These are both excellent informal phrases for certain situations.
The Situation: You see a person or people who is (are) doing something/completing a task/progressing in work, etc. and you want to tell him/her/them to continue. Maybe the person/group is having difficulty, maybe not; in either case you can use these phrases.
@Chris is correct because both of them mean the same thing: "Continue!". Now, for me (a male Amer. English native speaker about 24 years old), the difference that I see between them is only how or when I would pick one to use.
I would use "Keep it up" in almost any situation to speak to the person. Any time that I see that situation from above I could say "Keep it up". My opinion is that it is direct and personal to the person or group, and I am speaking to him/her about their own motivation and mentality. Example:
I am running 5 kilometers on the beach and I see someone ahead of me running with difficulty. If I pass him I can say "Keep it up!" (Continue running! Don't stop!)
I would more likely say "Keep going" (I would not use the "it" but you can) if I met people who had a job or plan or another "thing" that they were doing, so for me it is less personal about their motivation and more about their action. Examples:
In the street I meet two people who set themselves a challenge to walk 100 kilometers to ask for money to cure cancer. I would tell them: "Keep going" (Continue doing your challenge which is the action, nothing about the people.)
In my apartments, the fire alarm begins to ring at 10:00 p.m. I go out my door and meet my neighbour in the hall, but I remember that I should turn off the oven in my kitchen. While the alarm is ringing, I go inside for 15 seconds but I tell my neighbor "Keep going (Continue your action to exit). I will finish in a second."
I hope that this helps you to understand. I had to think about how I use these differently, and that is just me. Good question.