Hi there.
I feel guilty for saying bad things to him. This is correct. Guilt is feeling bad after doing something you feel is wrong.
I blamed him for my problems. This is correct. This means that you think your problems are his fault. Blaming is saying that someone is the cause of something.
It's his fault* that I feel like this. This means you feel he is the reason you feel like this. Fault is the responsibility for an action; as in, it's his fault this occurred.
Examples for each:
Guilt (noun): I will prove his guilt. Translation: I will prove that he in fact caused this.
Guilt (verb): I will guilt him into making me feel better. Translation: I will make him feel bad for what he did so he will be nicer to me (thus making me feel better).
Guilty (noun): He is guilty. Translation: he did this.
Blame (verb): I blamed him for this. Translation: I said he was responsible for this.
Blame (noun): He took the blame for this. Translation: he took responsibility for this.
Fault (verb): My friend said I cannot fault him for his actions, but I say otherwise. Translation: my friend says I cannot say he did something wrong, but I disagree.
Fault (noun): He is at fault for this. Translation: he is responsible for this.
There are also other definitions and usages for fault, but they are unrelated to what we are talking about here. If you're still interested, here are the other usages...
Fault (noun): His fault was not taking responsibility at first. Translation: his flaw is not taking responsibility for his actions.
Fault (noun): His action was a great fault. Translation: his action was a bad misdeed.
Fault (noun): There is a fault in the rock. Translation: there is an extended break in the rock.
Fault (verb): The rock was faulted. Translation: the rock was further broken, resulting in faults.