Both are correct, but the version with 'to' suggests a more formal occasion.
We might say, for example, 'He took us out to lunch', if this were a formal kind of lunch. But you would say 'We went out for breakfast/brunch/coffee' because these are informal outings. I don't think that we would use 'to' with these nouns.
If in doubt, use 'for', as it can be used for any kind of outing of this type, e.g. 'Do you want to go for a drink?', 'Let's go for a pizza after the show' and so on.