Usually, there's not a significant difference between the two. Your definition is quite accurate for both: violent anger, or extreme anger. With this meaning, "rage" or "fury" describe the emotion. To be clear, they can be used for situations that don't literally involve physical violence.
EXAMPLE:
He was overcome with rage/fury and screamed at the waiter.
One subtle difference, is that fury is often used to describe the anger of God or a natural disaster (i.e. an "act of God"), hence the idiom "the fury of a thousand winds."
EXAMPLES:
The hurricane struck with such fury that the small beach town was destroyed.
She was worried to face the fury of her god.