Both sentences are grammatically correct, but the choice between them depends on what "it" or "they" refers to. Here's the analysis:
(1) "I tried telling a couple of jokes, but they didn’t come off."
"They" refers to the jokes themselves.
This sentence emphasizes that the jokes you told didn’t succeed—they failed to land or weren’t funny.
(2) "I tried telling a couple of jokes, but it didn’t come off."
"It" refers to the act or attempt of telling jokes as a single effort.
This sentence suggests that your attempt at humor as a whole didn’t succeed.
Which is better?
If you want to focus on the jokes individually, (1) is more natural.
If you're emphasizing your effort or attempt as a single action, (2) is better.
Preferred in natural conversation:
Sentence (1) is slightly more common, as people often think of jokes individually in this context. However, both are valid and convey a similar idea with a nuanced difference.