We use come to describe movement between the speaker and listener, and movement from another place to the place where the speaker or listener is. We usually use go to talk about movement from where the speaker or listener is to another place.
When we talk about another person (someone who is neither the speaker nor the listener), we can use either come or go, depending on whether the speaker sees things from the receiver’s viewpoint (come) or the doer’s viewpoint (go).
[doer]
Yolanda came to
[receiver]
her mother for help.
We use come when we see things from the receiver’s viewpoint (in this case the mother).
[doer]
Melissa went to
[receiver]
her mother for help.
We use go when we see things from the doer’s viewpoint (in this case Melissa).
This is taken from the Cambridge Dictionary.
So using this explanation ( which I agree with as a native English speaker whose English is derived from Britain), I would say, " Can I come your office tomorrow" I say this because I am the speaker travelling to the listener's office.
However, If I ( the attorney) am speaking to the listener( a client) and we need to travel to someone else's office ( the real estate agent) I would then say, " We need to go to Peter's office tomorrow at 900 am for our meeting".
Peter if he spoke to either me or my client would then say, " Please come to my office at 900am tomorrow"
I hope this clears it up for you.