She tried to open the door but couldn't.

Literal

She [topic-は] door [object-を] try-to-open did [but-が] no-good was.

~(よ)うとする ('try to do, attempt to') uses the volitional form (here 開けよう) plus the quotative-like と plus する — literally 'do as if intending to open.' It marks an attempt that may or may not succeed. だめだった ('was no good') is a casual, blunt evaluative — the door wouldn't budge. The clause-final が ('but') joins the attempt with its failure. The whole construction is a stock pattern for thwarted action: tried + didn't work.