I can see why you are confused. But bear in mind that 'Got to go' is not a full sentence, and 'got' is not used as a verb in this expression.
As an informal way of saying that you need to leave now, 'Got to go' is a truncated form of 'I HAVE got TO go'. The main verb is 'have (to)' - meaning need or must.
Originally the word 'got' came from the verb to 'get', but now it is a kind of 'fossil', unconnected to its original meaning. 'Got' is an earlier form of the past participle of the verb get, still in use British English but no longer used in US English. In this sentence, however, the word 'got' has no meaning - just as 'got' has no meaning when we say 'We've got a problem'. In spoken English and in informal written English, 've got' and 's got' are simply alternatives to 'have' and 'has'.
I hope that makes it clearer, and that you can now see why we don't say 'Get to go' - as a phrase on its own, it doesn't mean anything at all.