The literal meaning refers to artwork. First, you draw a picture. The picture is made of lines (probably pencil or pen), with no color. When you color the picture, you color "inside the lines" so that the picture looks nice. If you color a tree with a green marker, you keep the green color inside the lines that you drew to make the tree. If you color "outside the lines," you might color part of the sky, for example, and then the sky will be partly green.
The term is often used literally in relation to children's coloring books. In a coloring book, there are black and white pictures, and children are supposed to color them. They learn how to color "inside the lines," so that each part of the picture is a single color.
As James says, "coloring outside the lines" means not conforming to rules. It could be positive or negative. If a child decides to color outside the lines in a coloring book, it could be a bad thing (if the child is simply being careless or defiant), but it could be a good thing (if the child is showing creativity and trying to add some original touches to the artwork).