It really depends on the context. "Check it" would be used if you wanted to verify if something was correct or true, or if you need to take a look to see the status of something and make sure it's okay. ("This fact in your essay looks wrong. Can you check it, please?" or "The saucepan sounds like it's boiling over, let me just check it.")
"Check it out" is used a bit more casually, and often in the context of seeing what something is about. "There's a fair in town! Let's check it out!". It's also used as an invitation to take a look at something that is enjoyable, cool or overall positive. "I just bought a brand new skateboard! Check it out!".
If "check it out" is not used in this casual context, it will be often used to indicate that something is in need of investigation or further information. "The crime happened on Belvoir Road, can you check it out please?"
I'm not sure if I've ever heard anyone say "check it up" before. I believe it might be grammatically correct, and you might use it if you were verifying if something was true or not, but I don't think people instinctively say it like that. My instinct would be to say "look it up" or just "check it/that" in that situation. ("I think Australia is a continent but I'm not sure, let me look it up." or "I think Australia is a continent but I'm not sure, let me just check that".)
I hope this is a little helpful; I'm not a teacher, just a native speaker, and I hope I clarified the phrases in the contexts you meant. I would be interested to know a teacher's opinion on "check it up" - the phrase just feels very unnatural to me.