My father allowed me to go swimming.

Literal

Father [topic-は] me swimming [to-に] go [nominalizer-のを] permitted.

許す (to permit) takes のを. The missing を particle after 私 is natural in casual speech.