'Instant' can be a noun or an adjective. An instant (noun) is a very short measure of time: 'It happened in an instant' means it happened very quickly.
'Instantaneous' is only an adjective. It's never used as a noun.
'Instant' & 'instantaneous' as adjectives have very similar meanings. They both imply that something happened very quickly or immediately, although I think 'instantaneous' feels a little quicker than 'instant'. i.e. He sent me an instant reply / he sent me an instantaneous reply - in both these cases the reply was very quick, but the second one feels slightly quicker.
You could also say 'he replied instantaneously'. In this case instantaneous with '-ly' on the end is an adverb. You can do the same with instant: 'He replied instantly'. Again, instantaneously feels slightly quicker.
I hope that makes sense :-)