'You got me' Can mean a number of things. It can be as you are playing a game, whether physical or a board game, and your opponent gets a score, going against something you did perhaps. It can also mean you do not know when a person asks you a question.