I can't figure out what the author is trying to say.

Literal

Author [subject-が] what [object-を] saying [try-ようとしている] [question-のか] don't-understand.

何を言おうとしているのか is an embedded question — 'what [the author] is trying to say.' ようとしている uses the volitional + とする for ongoing attempt. のか marks it as an embedded question that functions as the object of わからない.