When something is "on the verge" of something, it's almost reaching that level - similar to saying "on the edge". "On the verge of coherence" means it could almost be understood. When you hear people murmuring, you can't really understand the words, but if it "trembled on the verge of coherence" you could almost make out the words, but not quite. It's another metaphor like the lawn that ran up the beach - it's written to evoke the feeling of waves in the ocean. You're going to encounter a lot of interesting English in The Great Gatsby! F. Scott Fitzgerald was the quintessential American writer of the 1920s, and that was the same time period that American English Slang and popular culture really exploded.

Verge: the extreme edge, rim or margin of sth The point beyond which an action, state or condition is likely to begin or occur “The whole book had thrilled me to the verge of spiritual exhaustion.” "The murmur trembled on the verge of coherence” = The low, indistinct sound was barely audible (on the verge of not being audible) 她的衣服是深蓝色的，接近于黑色 Her dress was a very dark blue verging on (close to) black. 他挥霍掉了大部分钱财，濒于破产。 He squandered most of his money and was on the brink of (on the verge of) bankruptcy. 三天之后他在自己的房间里被发现，已濒于死亡。 He was found three days later on the edge of (on the verge of) death in his room.