"That's all that matters" means regardless of anything else, it can mean a willingness to ignore any bad/weak qualities.
For example if the person was very shy and didn't know how to communicate well the other person could say "You know you're a genius right?" This would be a confidence booster. Followed by the phrase "That's all that matters right" meaning that the person didn't mind that they couldn't communicate well as their good qualities overcame their bad qualities.