Yes, you've got the right idea. There's no difference in meaning. The "to" before "go" can also serve as the particle for "to wait" or you can put another "to" in front of "wait."
One "to" can serve as the particle for multiple verbs:
I like to run, swim, lift weights and play basketball.
You could put "to" in front of every infinitive verb ("to swim", "to lift weights", "to play basketball"), but it makes your speech or writing less repetitive when you just use one "to" to serve as the particle for a series of infinitives.