"go out doing something" is like "go out while doing something", "go out in a certain manner" described by the specific 'something' in the sentence.
"go out to do something" is like "go out in order to do something", "go out for the purpose of doing something".
Note that in the first construction "go out" can be a literal going out or a figurative (idiomatic) one.
Literally, "The detective was so angry that he went out muttering to himself": He left the premisses; he was muttering to himself at the time. "to go out" = "to leave", "to exit"
Figuratively, "The team won the final game of the season. It was quite a thrill (for them) to go out winning": They finished/ended the season as winners. "to go out" = "to finish (a season, a career, one's life)".
Also, again figuratively, "The poor guy finally died of cancer, but at least he didn't go out whining about the injustice of life": He didn't die (bitter and) whining. "to go out" = "to die".
The second construction is normally literal: "Let's go out to get some fresh air". "They've often gone out to get ice cream at midnight".