My father signaled me with his hand to leave the room.

Literal

Father [topic-は] me [to-に] room [object-を] exit [so that-ようにと] hand [with-で] signaled.

~ようにと marks an indirect command: 'go so that you leave.' 手で合図した (signaled with his hand) uses instrumental で.