To be 'caught bang to rights' is informal British English. It's equivalent to the more widely-known idiom 'caught red-handed'. Both mean to be caught either in the act of a crime, or with unquestionable evidence to connect you to the crime.
Not everyone in the UK uses the expression, but everyone understands it. You often hear it in the dialogue of police dramas set in England for example, particularly those involving more working class characters.
The word order also sounds natural in colloquial British English, particularly some regional dialects. Wallis is from Lancashire in the north of England, and the emphatic sentence structure 'Caught bang to rights, you were.' (with a pause and a comma) fits his character perfectly.
Finally, the 'us' means the two of them. Wallis and Gromit are very much a team.
I hope you enjoy Wallis and Gromit films. They're works of genius in their modest way.