"back talk" is basically speech which is specifically meant to go against something the listener has said. It is almost always used against someone in a position of authority or power over the person doing the "back talk". Usually it is meant to change the mind of the person in power so that the speaker of the back talk can avoid something unpleasant. It may also be simply to vent frustration or to simply refuse.
For example, a teenager might give back talk to his parents when they ask him to do chores. He might say "I just did the dishes last night, and there aren't very many dirty ones." He hopes that he can get his parents to agree that he does not need to do those chores.
If your boss asked you to do something and you say "No, I quit," that would be an extreme form of back talk.