If you "just remembered", A1 and A2 (Simple Past) are usually going to be the right choices. The only time you need to use the Past Perfect (like in B1 and B2) is when you're saying that the action was completed later than some other action in the past, or was completed after another point in time in the past; and that doesn't seem to be the case in either example. When I remembered what you had done to me, it made me mad. Six months later, I remembered what you had done to me. When I remembered that you had broken into the headmaster's office, I decided to change the subject. I remembered that you had broken into the headmaster's office just six days before you were supposed to graduate.

In most cases, the A choices are what would be needed. However, in certain contexts, the B choices would be better.

A.1 and A.2 sound natural because they use simple past, which is common in conversation. B.1 and B.2 are correct but use the past perfect, which emphasizes that the action happened before the act of remembering. They're more formal and less conversational, but fine if you want to highlight the sequence of events.

Only A1 and A2 sound natural. B1 and B2 sound very awkward - the only way they can possibly work is if speaking extremely slow and formal, as if reading a poem or a brief statement at a funeral ceremony. Normally, even if we want to use "you had" in these sentences, we would abbreviate it to "you'd", to make it sound a bit less of a mouthful - a bit less awkwardly wordy & bookish. Also, to make the sentence more balanced, if you're using "you had done" or "you had broken" in the last half, you should also consider using "I had just remembered" or "I have just remembered" in the first half, to maintain a consistent tone.