I'm not offended by the word gaijin, I use it myself too, referring to myself or my other gaijin friends. Sometimes Japanese people too, since to me THEY are gaijin ^^
I agree with all the comments above however, there are people who have negative/narrow minded views about gaijin in Japan. They usually make very wide generalizations, and what you do new falls to the same category you're in. Most of the people I met in the university I did study abroad started to think my personal habits/choices are general truths for people from my country.... And they're university students who study English, not uneducated ignorant people. What I mean is, there are such people and I would find them offensive even when they say "gaikokujin" or "gaikoku no kata" or whatever.
But the opposite is also true, I met lots of very open minded Japanese people who used the word "gaijin" but never meant any offense.