I understand what you want to say, but it's a bit unnatural and feels like a strange question to ask. Is it about being on a date and someone is looking at everyone else other than you?
If you could provide context as to why you're saying it, I might be able to give a better phrase.
Correct me if I'm mistaken, but you want to say: Where this person is looking, or their facial expression is making it clear to you they are uncomfortable or unhappy with whatever is happening.
In that case, the word "belie" is really useful. It can be used to show contrast, so even if someone says they want to do something, maybe their body language says otherwise.
"Your eyes belie your words"= you say, "yes" but your eyes tell me: no.
In general, I would just say, "It seems like you'd prefer to leave, why?" or "Why don't you want to be here (with me)?"