'The' is used when the specific physical building is meant.
1. I went to the school to talk to my son's teacher.
2. I live quite near the hospital so I often hear the sound of ambulances.
3. Her dad works at the prison.
In the above sentences, the specific building is important.
We do not use 'the' when we are using these words to describe a state or fact about the person but the physical building is unimportant.
1. Children go to school at age 5 in the UK = when this stage of life starts
2. She was quite seriously ill so she went to hospital = her state was serious enough for hospital treatment.
3. Her dad is in prison = he has been convincted of a crime.
The specific building is not important in the above cases.
We only say, 'the gym'. I suppose we consider that, in this case, we always mean a specific building.