A recoil is a sudden movement backwards. Recoil also carries the sense that this sudden movement back was because of something you did or someone else did and that this action was (usually) not positive.
"I recoiled in fear at the sight of the snake"
'recoil' carries 2 ideas:
1. I moved suddenly back
2. I moved suddenly back because of something not positive (sight and fear of the snake)
In your example, "Violence does, in truth, recoil upon the violent" (Arthur Conan Doyle).
Here, the act of violence snaps back/bounces back/returns to the 'violent' (the violent person/the perpetrator of the violence).
Again there are 2 points being made:
1. The violence is suddenly moving back to the 'violent' (the person who makes the violent action)
2. This sudden return (of violence) is a caused by something negative - the initial violent act.