It's your responsibility to decide whether we go there or not.

Literal

We [subject-が] there [target-へ] go [or-not-かどうか] [object-を] decide [nominalizer-のは] your responsibility [copula-だ].

Two important structures. かどうか means 'whether or not' — it's the embedded-question particle か followed by どうか ('or how'). The whole construction decides-or-not is then nominalized by の (the other main nominalizer besides こと) and marked as the topic with は. The structure [clause]+のは[X]だ is the standard 'it is X that [clause]' pattern.