It is an angry remark. Consider this example: "With a spurt of fury Keith snatched at the letters and threw them on the ground. 'To hell with you!' he shouted, his face empurpled. 'You're fired!'" Keith is furious at someone. Keith has reached his limit. Keith will not talk to this person any more. Keith wants this person gone. Not only does he want this person gone, but he wants them to go "to hell." It is just an expression. Nobody thinks about the literal meaning. However, the literal meaning is "I want you dead." It is worse than "I want you dead." "I don't just want you dead, I want you dead and in hell." In the Western tradition "hell" is the afterlife where bad people go, where they suffer terrible pain forever. "The hell with X" always carries the idea of anger--or at least irritation. It carries the idea of "no more talk." It carries the idea of "I've had enough." When said to a person, it means "Go away. Don't speak to me any more about this." It can also be said about a thing. "The hell with that." "Should we go to a movie?" "It's a long drive. And parking is expensive. And we'd have to get a babysitter. And I'm tired. The hell with that, let's just stay home and watch TV."

Matt and Tim are right. Just for further information, "to hell with [X]" can also be used for cursing or dismissing other things or ideas, not just people. For example: A: "I don't think we should get married. It could ruin your career." B: "To hell with my career. It's not important." You can also just say "to hell with it" as a general way of giving up on a situation, a plan, etc.: "I can't get this key to work. To hell with it, I'd better just break the lock." "I waited for you until midnight, but when you still didn't show up, I finally decided, to hell with it, I'm going to bed."

As Matt said “To hell with you” means go to hell. The verb is implied, similar to the way someone might say “Off with you” to mean “Go away”. They both sound more British than American to me. And as he said, no, it’s definitely not the same as “I’ll gladly go to hell with you”

@Matt J // Thanks!!

Ill gladly go to hell with you means .. I will never leave you, I trust you and will always stay with you. To hell with you - is bad, it is telling someone go to hell, I don't like you ,