I think a third sentence should be added to your list:
1) Don't be afraid of failing
2) Don't be afraid to fail
3) Don't be afraid of failure
I will only describe what I perceive to be the differences in meaning in one very specific situation. A boy is about to take a test. He is scared because he feels he might not pass. His father speaks to him. The father can say any of the sentences 1-3 but I would interpret them differently:
I would hear #1 as an attempt to reassure the boy that he will not fail this particular test. Of the three sentences, this is the most optimistic.
With #2, I hear the father tell the boy to accept the possibility of failing, but without fear.
#3 is similar to #2, but more is more universal in that the father speaks of failure itself, and not just in reference to the upcoming test. #3 is a request that he be a courageous person.