"En absoluto" may also be used as an affirmative answer. Although is not that common. I normally hear it more commonly used as a negative answer. ("not at all")
I am a native myself and tend to get the same expression mixed up, normally context and the way people reply help out to understand whether they say yes or no.