Purgatory is an idea from the Catholic doctrine wherein after they die, people who are neither virtuous nor sinners are trapped in a middle ground between heaven and hell. As there is nothing there, they can neither sin nor do good but are rather confined to stay in this place.
In English the meaning has spread to being used when someone finds themselves in a situation where they are 'between a rock and a hard place' or don't know what to do. If I am damned if I do and damned if I don't, then I am in purgatory, figuratively speaking. It's quite hard to explain without using other idioms, but basically finding oneself in a situation where either choice is equally undesirable.