They both mean it's not working, but "it's broken" is the correct usage.
"It's broke" while technically incorrect, is still commonly used in informal conversational English. But if you want to speak in a correct or more formal register, then you would avoid using this form.
When trying to choose between 'broke' or 'broken' (as an adjective), the general rule is 'broke' should only be used when describing being out of money.
"He is broke" (He has no money/He has run out of money) is correct, for example.
For all other cases, use 'broken' as the adjective.
"My watch is broken"
"My arm is broken"
"Those toys are all broken".
"He is broken after the breakup with his girlfriend".