No, you can't. 'Families' means two or more separate families - for example, "The students all went home to their families for the holidays".
'Family' refers to a group of people, not individuals, just like 'team' or 'committee' or 'staff'. The individual players, for example, are not in themselves 'teams' - they are 'team members'. This error is common among Chinese learners. Is this because of first language interference, I wonder?
It's fine to say " with my family ".