It all depends where the speaker and the listener are in relation to each other.
'Come' indicates a movement TOWARDS the speaker.
'Go' indicates a movement AWAY FROM the speaker.
Let's say that you want to enter the doctor's room. You are waiting at the receptionist's desk, outside the room.
The receptionist would say 'Go in', because your movement would be away from this speaker.
The doctor would say 'Come in', because your movement would be towards this speaker.
Simple, isn't it?