'To suit' and 'to match' are not synonyms. I would say that here it isn't possible to use 'match.' When you say 'it matches me' you are saying it matches the colour of you (the colour of your body.) That is a strange thing to say.
You could say something like 'The colour of this skirt doesn't match my personality.' For example, if a person has a quick temper then you could say that a red skirt matches their personality, given red is the colour associated with anger. But this is a bit of an odd idea as well.
Saying an item of clothing 'suits someone' means that the clothing looks good on them. If something doesn't suit someone then the opposite applies.