Yeah I think he's making a joke here because 'pith' sounds like 'p*ss'. Sounds like he's actually using an orange or lemon in his recipe, so he's literally taking/using the pith from the fruit but also joking about taking the p*ss out of someone/something.
This sort of 'joke' - one that makes sense but isn't really that funny - is sometimes (in England anyway) called a 'dad joke', because it's the sort of joke your dad might make. Rick Stein (the chef talking in your example) makes a lot of dad jokes :)