I can understand your confusion, but this response is correct, normal and - most importantly - absolutely instinctive to native speakers. Please don't make the mistake of thinking that the second person's response means "You are wrong", as it might do in Chinese. The "No" is is NOT a comment on accuracy of the other person's belief - it is an echo of their negative statement.
Look at it this way. Are you familiar with short answer echo questions, for example, "Yes, I am" , "No, he didn't" or "No, it wasn't" ? The words "Yes" and "No" have to agree with the statement, don't they? If the statement is grammatically affirmative, it is preceded by "Yes"; if the statement is grammatically negative, it is preceded by "No".
Well, this response it just the same. The second person is saying "No, he didn't[want to leave]". The "No" belongs to the negative statement it is attached to. The speaker is confirming what they have heard by echoing it. If the statement is negative, you use "No"; if it's positive, you use "Yes".
I remember a case when one of my Chinese students here in Britain got into trouble because he hadn't understood this point. He was living with an English family, and one Saturday morning he told his hostess that he was going to London for the day. She replied by saying "So, you won't be home for dinner, then?", and the student simply replied "Yes". The hostess interpreted this affirmative answer as "Yes, I will be home for dinner". So she made him dinner, waited, waited ...and was fairly angry when he returned home at midnight, having eaten in London.
So, as you can see, it is important for you to get used to this key difference between Chinese and English. "No" doesn't mean "Yes"... it means "No".