They are different pasts.
"I was playing the piano" is a past continuous. Is used when the action you are talking about was not finished yet when, for example, something happened: I was playing the piano when the bell rang.
"I used to play the piano" is something you used to do in the past, but you don't do it anymore. Ex. I used to play the piano when I was a kid.
Also, this usually refers to a habit, like playing a sport, playing an instrument, or going in a certain place (ex. I used to go to school by bus every morning)
"I had played the piano" is a past perfect. It means that you did it before something else (ex. I knew her because I had played with her years before).
"I had been playing the piano" is a past perfect progressive. It is something that happened in the past and ended in the past. (Ex. The boys came home very tired because they had been playing football)