This is difficult to explain, but I'll try. In this case, 'game' is used as a metaphor for something else. In the literal sense, if you join a game late, you may be at a disadvantage because points may have already been scored, etc. You can take that meaning and use it to describe real situations such as careers, education, etc... i.e. "He started his finance career so late in the game." This means the person entered this career really late and may have some disadvantages as compared to those who started earlier.

It can mean that you also waited until last minute to do something. For instance, "I can't believe he wants to change the floor plan of the house we are building for him this/so late in the game!" It means that this person should have made changes way earlier, instead of making last minute alterations. :)

The meaning could depend a bit on the context. In a baseball game, one could ask why the manager only put in a competent player '"so late in the game''. The player didn't have a fighting chance to help the team. In a open ended situation, ''so late in the game'' could refer to backing down after stalling for no reason. He held the property for decades and never build the shopping center he advertised. When he finally sold the property to another developer people wondered why he sold so late in the game.

It would be easier to explain if you gave us the sentence where you found it. "So late in the game" has both a literal and a figurative meaning.