I haven't heard this phrase before, but here's how I interpret the sentence: The author seems to be saying that the freaks and ghouls are at a lower level in society, and the subject of the sentence, "he," spends time with them down there. The "down" in this sentence may also be about physical location. For instance, in the wider context of the paragraph, maybe the people referred to as freaks and ghouls frequent "downtown," the center of town, and in this case the subject of the sentence "he" could hang down with them (downtown), even without the author feeling morally superior to anyone. I don't think "hang down" and "hand down" mean the same thing. "Hand down" is a concept in American English that means to give something of yours down to somebody else as a next owner. For example, an older brother might hand down his shirt to his younger brother when the older brother grows too big for the shirt. But "Hang down" isn't a common phrase as far as I know.

I've never seen “hung down.” “Hand down” is usually limited to parents giving things to their kids when the parents die (example: “The diamond necklace has been handed down for generations in her family’”). Hang out means to spend time together, usually with friends and usually just for talking rather than any specific activity.

no down is not connected with hung in that sentence it is not like the phrase "hang out" it means literally "down" with the freaks etc. the verb to hang means to spend time with to relax with ie. I hang with my friends/ today I hung with my friends I can see why that would be tricky though the usage of down there would be like : today I hung "down in the basement" with my friends. for example

So, this is really slang, so not a good idea to copy . To hang out with people is to spend time with them. In the example it is shortened to just "hang" instead of "hang out". The word "down" here is not part of a phrasal verb, it belongs to the freaks and ghouls, who probably live down by the swamp or down by the railway station. To hand down means to give things that you no longer need to, for example, a younger member of the family, like an older brother hands down his toys to the younger brother.