" I don't get it" is a casual way of saying that you don't understand.
If you don't understand a difficult subject such as math you could say "I don't get math"
Be careful though, if someone says "I don't get cold" or "I don't get paid" they are referring to a lack of something not that they don't understand.
The difference is often in the context of how it is used. "I don't know" means that you don't understand.
This can be used in a similar way to "I don't get it" by adding words to the end such as "I don't know him" or "I don't know math".
It will always mean you don't understand.
If someone asks you a question you don't know the answer to you should use "I don't know" if someone says something you don't understand you can say "I don't get it" OR "I don't know what you said"