Howdy Yoichi!
No, this would not be a natural way to describe the situation you experienced. “Chill” is usually used to describe times of relaxation or ease. It generally has a positive connotation. For example “We didn’t do any sightseeing on our trip, we just chilled by the pool”. For your experience I would perhaps say something like: “He was in no hurry” or “He took his own sweet time”.