'I am a mess' refers to you personally. This could mean physically or emotionally. If you are dirty, dishevelled and untidy, you could say that you are mess. Likewise, if you are emotionally 'screwed up', you could say that you are a mess. This means that YOU are the mess.
'I am in a mess' means that you are in a difficult situation. There is a mess - a problem, a confusion - and you are in the middle of it.