Hi Jin! That is one of those difficult issues in Spanish which has no exact rule :( haha. But I'm going to try to explain it in case it's useful for you or other students
"Ir" means "to go", so you could always use it and be understood. You should use it in general contexts, eg: "Voy al colegio todos los días" (I go to school everyday)
"Irse" feels more like "leaving" as you've been told. As a Spanish native, all the examples I can think of with "irse" are "Me voy a dormir" (I'm off to sleep), "Me voy a casa" (I'm going home). So you use those when you tell somebody you are leaving that place to go somewhere else.
Generally, you could always use "ir" and never be wrong. All those sentences with "me voy" could work the same way with just "voy" (Voy a dormir, voy a casa). So if you have doubts, don't use the reflexive form.
Now, as a native speaker, what's the difference I find between:
"Tengo que ir a casa"
"Tengo que irme a casa"
They are both very similar and could mean the same, but if you want to be picky, I would use the first one to say "I have to go home to do ____ (whatever) and then I'll be back (or then I'll do something else)". I wouldn't definitely use it to say "goodbye, I'm leaving". It sounds odd.
But if you say "Tengo que irme a casa", you could mean hey, it's late so I'm leaving, see you tomorrow.
Sorry for the looong explanation! It made me think as well haha :)