The OP, I guess it's time for you to lead a revolution of the Chinese language because 老外 is definitely the most popular colloquial name that's used to refer to foreigners currently, of course it's no problem talking about this topic with non-Chinese speakers, as you expressed, linguistically, if you will. But I thought you were asking 'if it is offensive today', now you seem to switch to 'where does it derive from', sorry, no idea.
Sir Woz, glad to know you finally admit that it's not offensive to Chinese ears. As I said before, let's pretend the English word 'foreigner' is a very polite word in English, then can I, brat 一一一, as a beginner of English, assert that oh, what a stupid word, how obscene English is, they call me such a dirty word - foreigner! They have no manners!
The question is, am I qualified?