You're right both can mean to visit somewhere: for example, 'why don't you come around to my place later?', or 'can you come over to my house later?'. In that context they are literally the same and completely interchangeable. However to 'come around' also has another meaning, which is to change one's opinion and see something a different way: for example, he finally came around to liking spinach after years of me telling him it was good.
Hope that helped buddy :)