Just to add to what the others have said, looking at the grammar of the phrase...
The expression is always '..... a hell of A...' . Note the word 'a' at the end. The expression has to end with an indefinite article, followed by a singular countable noun. That is why your suggested expression 'a hell of trouble' was wrong. 'Trouble' is an uncountable noun, used without an article, so it can't follow directly after 'a hell of'. Paul solved your problem by changing it to 'a hell of A LOT of trouble.'
If you look at all the examples which Paul and Jonathan have suggested, they are all with singular countable nouns ( .. a long way ... risky decision.. a time .. a man).
Finally, if you think that 'hell' might upset someone, you can always tone it down and use the less offensive 'heck' instead. "It was a heck of lot of trouble" is less likely to offend people's sensibilities.