Hi,
I would say that they're all correct. I think it's most common for "in vain" to come right after the verb, like in the first sentence, "They tried in vain to open the locked door."
The least common and most formal are the sentences where "but in vain" appears at the end. ("They endeavored to make her happy, but in vain.") You would probably only find it used that way in formal written language, like in a novel.