There is no simple answer to this. What native speakers say--even educated native speakers--doesn't fit any logical rules of grammar. The same problem occurs in an even stronger form with "feel good" and "feel well." "I feel good" means "I am happy, joyous, optimistic, energetic." "I feel well" means "I feel healthy, I am not sick."
In a formal situation, where you want to certain you are using grammar, the only solution is to express your meaning in some completely different way. This is the answer to many difficult usage questions: rewrite the sentence and sidestep the issue. For example, "I am sorry about X" instead of "I feel bad about X."
The American Heritage Dictionary offers a usage note: "The adverb badly is often used after verbs such as 'feel,' as in 'I felt badly about the whole affair.' This usage bears analogy to the use of other adverbs with feel, such as 'strongly' in 'We feel strongly about this issue.' Some people prefer to maintain a distinction between 'feel badly' and 'feel bad,' restricting the former to emotional distress and using the latter to cover physical ailments; however, this distinction is not universally observed, so 'feel badly' should be used in a context that makes its meaning clear. · 'Badly is used in some regions to mean 'unwell,' as in 'He was looking badly after the accident.' 'Poorly' is also used in this way. · Note that 'badly' is required following 'look' when it modifies another word or phrase in the predicate, as in 'The motorcycle looked badly in need of repair.'"