"I am afraid that I cannot help." - is correct. It means "I regret that I cannot help."
"I am afraid if I can help." - This is incorrect and would never be used. The nearest meaning would be:
I am afraid that I might be able to help.
If this has any meaning, it is the exact opposite of the first statement.