They can each be used when you don't want to tell someone something, but "it doesn't matter" comes off a little more passive aggressively. It doesn't matter can also be used to mean its literal meaning in which case it can NOT be substituted for nevermind. Habría escrito esto en español, pero no habría sabido como!