These words are very difficult to use, Actually I think most of the japanese don't use it in the strict sence.
As you say 暖かい is an oposite to 寒い、温かい is an opposite to 冷たい, but how you express the situation 寒い or 冷たい depends on how you 'feel'. When you are blown by the wind after the bath, if you are hot enough, you will feel it as a "cool breeze" but if not, you might say as a 'cold wind'. That means you can express the situation in both ways, but it depends on how you will feel.
So I suggest you make distinction as just what you said(decide according to thinking the contrary), and don't worry too much. Whatever you use, most of the time it is correct.
『暖かい』をよく使う場合:空気、衣服、布団、心
『温かい』をよく使う場合:料理、飲み物、対応、態度
両方使う:気持ち