They basically mean the same but there might be some differences in the contexts where they are typically used.
If I would have to explain who to make a difference between those two, well, to me "custom" is more like a habit, something you don't really think about you just do it. "Tradition", on the other hand, is something with a deeper meaning, possibly religious, cultural or family-specific that guides the person's/groups way of behaviour or performance in certain situations. But these are just my opinions.
According to CambridgeDictionaries Online:
Custom = a way of behaving or a belief which has been established for a long time;
a local/ancient custom
Tradition = a belief, principle or way of acting which people in a particular society or group have continued to follow for a long time