You will never hear a native speaker use it in the context of the morning or afternoon. "Let's call it a night (or day)" means to stop something for the rest of the night (or day). A sample sentence: Let's play one more hand of cards and then let's call it a night. Interestingly, the original phrase was "call it half a day", first recorded in 1838, which referred to leaving one's place of employment before the work day was over.