It is common to hear, but I never say this. People say this when they are trying to give a balanced description of a person or a situation. For example, "In all fairness, although he is an amateur, even a professional would have failed in that situation." "In all fairness, the chef's recipes are popular, although I don't life that type of food myself." "In all fairness, I should have warned you that I was not going to be available, but you still should have been prepared." It is something like admitting a good point about the other side. Good luck!

“In all fairness, that was actually really rude of her to say.” “In all fairness, you shouldn’t speak to your sister like that. She was only trying to be kind.” “In all fairness, it’s actually illegal to drink in public here. You should probably go home with your beer.” A phrase used to express a contrary point of view, most often when someone says something crappy and wants validation for their bad behaviors. It’s a polite way to correct someone and say you don’t agree with what they said/did.

Bob is a total jerk. He’s mean to everyone. But, in all fairness, he was right about the gas leak and he saved a lot of people. Very common!

I agree with the other answers and I will add something more. It is not nice to speak unkindly about someone. If you need to use this phrase then you are probably saying something unkind. I would try not to use this phrase because I will try not to speak about someone in this way.

Others have given good examples of "in all fairness". It's pretty common. However, I would not normally use 'just to be fair' in those situations. It's rarer and something I'd say when talking about the fairness of a situation. Just to be fair, we shouldn't make a final decision until we've received all of the responses. (I'm suggesting that the final decision won't change, but fairness requires that we wait.) Some people might use 'just to be fair' to mean 'in all fairness' but I'd say that it's generally weaker.