Personally, I think that the connotation of the word ‘elusive’ depends on how you use it.
You can use it to express someone’s nature, meaning someone is hard to reach/contact/evasive, and for me that’s bad.
But when used in another context, for example if you don’t want to hurt someone else’s feeling, you can give them ‘an elusive answer’, meaning ambiguous/undefinable/indistinct, so it’s neutral and not offensive.
So, it depends on how you use it.
I hope this helps 😄