I assume you are talking about using all three words as a verb. The words "try" and "test" can also be used as nouns, but "prove" can only be a verb (the noun form of "to prove" is "proof").
"To prove" (verb) means to demonstrate (or to show) that something is true, and implies that evidence must is needed.
Examples:
I will show you a map and PROVE that we have traveled five miles.
Can you PROVE that your story is true?
A very common exchange in American English:
Person A makes a statement, for example, "My dog is smarter than your dog."
Person B challenges with, "Prove it."
"To try" (verb) means to make an effort to do something.
Examples:
I will try to do well on the test.
Please try to fix my car engine.
A common exchange in American English:
Person A, explaining why he has failed, may say, "I tried my best." (meaning: I gave it my best effort.)
Person B may reply, "Try harder." (meaning: you should be able to make a better effort, so make a better effort and succeed.)
"To test" means to check the ability or quality of something, perhaps to determine how fast a car can go, or how much a person knows.
Examples:
We will test the new race car on the track tomorrow.
I may test your knowledge of history.