Generally speaking, your explanation is correct. This is normally the way these phrases are used. However, as Cathy said, there is no distinction... there is no rule about it. If I said "when I was little..." I would be talking about my early childhood, the idea being that when we are children, we are not fully grown (big). However, if I said, "when I was young", I could be talking about the time of my youth (although, I would more likely say, "when I was a teenager"), my twenties or even my thirties, since I am in my sixties now. So, that phrase can be relative to the age of the speaker.