Hello is probably the most formal. If I was answering the phone at my office, I would say "Hello, this is Alice" but not really any of the others. I would usually say hello if I was answering my personal phone too, because I might not know who is calling.
Hi is a little informal, a little social.
Hey is more informal, and also it is something that only younger people say. Older people might be a little insulted, because they might think you should only say "hey" if you are shouting at someone. "Hey! Get off my lawn!"
Hiya is also informal, probably the same level of informal as "hey". The difference is that "hey" is a little more of a young person or aggressive style of speaking, in addition to being informal. People might say "hiya" but probably would not write it. I also think it is less common.