Good question. No, it doesn't refer to kidding (although that would be similar "don't fool with me" could have that meaning). "Don't make a fool out of me" means that you want someone to avoid damaging your reputation by his behavior with other people. This may refer to a situation where he is deliberately trying to make you look bad, or it might be that his or her foolish behavior ("making a fool of himself") could reflect onto you, due to your close relationship.
There's usually no significant difference between "don't make a fool out of me" and "don't make a fool of me."