If we know the biological gender of the dog or cat, we call them 'he' or 'she' according to whether they're actually male or female. The owners of dogs and cats will get quite offended if you call their pet 'it' - the neuter pronoun sounds very impersonal and unfriendly.
Bear in mind that we have no grammatical gender in English (only biological gender), so calling an animal 'it' makes it sound like an object, or a pest of some kind. Unlike some languages, we don't really even have words to distinguish between a male or female cat, dog, rabbit etc. ( and if we do, we don't use them in everyday conversation). While it's fine to call a spider or cockroach 'it', or even a mangy fox that the farmer wants to shoot, you should never call a family pet 'it'.
So, what do you do if you don't know whether the pet is a 'she' or a 'he'?
Well, let's say that you meet a neighbour walking his or her dog, and you want to make polite conversation referring to the dog. Basically, what you have to do is avoid using any pronoun at all until the person tells you whether the dog is male or female. So the conversation can go like this:
You: [Indicating the dog and smiling] Ooh, who's this, then?
Neighbour: This is Rosie. She's a great Dane/chihuahua cross.
You: [Breathing a sigh of relief that you now know what sex the dog is] She's lovely .... etc.
If you're less lucky, it might go like this:
You: [Indicating dog and smiling] Ooh, who's this, then?
Neighbour: This is Bailey.
You: [Thinking: Hmm...dammit. Still don't know] Bailey?
Neighbour: Yes, nice name, isn't it?
You: [Addressing the dog in desperation] Hello, Bailey. You're nice, aren't you? How old are you then?
Neighbour: He's six months old.
You: [Greatly relieved] He's lovely, etc....