"To go" = "remaining" here.
There is one hour to go = There is one hour remaining
We have one mile to go = We have one mile remaining
The sense is that the item (e.g., hour, mile) that is "to go" will be used up or passed through.
Do not confuse this with the expression "I'll have my food to go." Here "to go" means "I will take my food with me rather than eat it in the restaurant."