If you're referring to your place of work, we would usually say "I am at the office" because "the office" is referring to a more general location. In other words, you're simply saying, "I'm at the place where I work" - this could be at your desk, in a conference room, in the WC, or outside smoking a cigarette.
Sometimes, people have a room in their house that they designate the "office." If you are referring to a specific *room*, you would say "I am in the office," just as you would say "I am in the kitchen" or "I am in the bathroom."
You might also say this while at your work place if you are emphasizing your physical location in a specific room/space ("John, I can't talk now, I'm in the office" emphasizes you are *in* the office building, implying that people around you are working and don't want to listen to your phone conversation!).