"It doesn't matter." or "It does not matter." would be correct.
I wouldn't necessarily say that is spoken by a lesser class (I'm not downgrading your answer Jeff). It is true that it is not proper English but like all countries - we all have our accents or dialects. I can't speak for the United Kingdom but in America if you go to the South you will hear this a lot. I'm talking about Texas, Alabama, Georgia, Mississippi, Oklahoma, Louisiana, etc. People use it all over. For example, my grandparents (who have passed away now) and even my mom still occasionally uses it, and I find myself out of old habit catch myself still saying it here and then randomly when I'm talking fast because I grew up hearing it.