To be invited out of the blue, means you did not get invited early enough to make plans. It is like someone invited you at the last moment, and you do not feel really welcome.
If you attended a party, dinner, or event "out-of-the-blue", that means you were not invited, and it would be a surprise that you "showed up" or arrived, because you did not receive an invitation. You arrived "out of the blue"....or on the spur of the moment, and they did not expect you to come.