You were probably invited to go somewhere, but did not go.
This means that it was a pity you were not there. They would be really happy if you were there = they were missing you there = you were missed.
You don’t reply I missed you too.
You would reply.
I am sorry I could not be there….
I am sorry I was not able to come….
I wish I was there too, but unfortunately…….( Say the reason why…..)
Anything along those lines.