Either essentially means both but is more specific. "He doesn't speak to both of them" makes it sound like he does speak to them, just not both at once. "Either" means that he doesn't speak to them both at once or individually; he doesn't speak to them at all.
Neither means very much the same thing but the important difference is that it's a negative word, meaning that if you say "he doesn't speak to neither of them" that means he DOES speak to both of them.
A quick way to remember the difference is to think of it like this:
Either: both (at all).
Neither: NOT both (at all).
Edit: if you want to use neither in that sentence, you just need to change the wording slightly.
Either: "Peter has two brothers, but he doesn't speak to either of them."
Neither: "Peter has two brothers, but he speaks to neither of them."