The main difference between the two sentences lies in the tense.
"didn't have" -> simple past
"hadn't had" -> past perfect/pluperfect
The first sentence just describes a state in the past. Somebody could, for instance, say this about a person, that they had to work with in the past and now don't. Back then the relationship was not good and now it does not matter since they dont have to see each other anymore.
The second sentence alse describes a state in the past, but the pluperfect implies that there has been a more recent "past" in which the state (of your relationship) was different. This sentence, could be said in the context about a person that they didnt like in the beginning, but started to like at a certain point.
So your interpretation is right on the money. :)