"I called you, but you didn't answer the phone."
This is fine, but you don't need 'the phone'. It should be obvious from the context that 'called' means 'phoned', so it would be more normal to say just 'I called, but you didn't answer'.
Or if you do want to add '..phone', we would probably say 'your phone', presuming you called someone's mobile phone/cellphone. 'Answer the phone' suggests a fixed line phone in a building.
"I called you by phone, but nobody picked up."
It's correct, but a little unnatural. The phrase 'by phone' is redundant.
"I tried to contact you on the phone, but nobody answered."
'On the phone' is used when someone is in the process of having a phone conversation. If you're referring to the means of communication, 'by phone' would be more normal.
"I tried to reach you by phone, but nobody answered."
This is OK, but note that the phrase 'nobody answered' or 'nobody picked up' suggests that you are phoning a landline in a location such as an office, or perhaps a family home. It suggests that there are a number of different people who might have answered. You wouldn't say this if you'd phoned the person's mobile. If you're calling someone's mobile, you'd say 'You didn't answer' or 'I got no answer' or 'It went straight to voicemail', for example.
"I called you by phone, but I can't reach someone".
No. As with the sentences above, 'I called you by phone' is something of a tautology, and 'I can't reach someone' doesn't make much sense. Forget this one.