The first example is correct as you have to use the gerund '-ing' --- It's the standard [go + ing] form ''go swimming''.
"go to swim" is however possible but, there's something missing. You might have to emphasize a location to make it work so, the sentence would be ''go (to the beach) to swim''.