If the pen's travel is too short or its movement too slow, the gesture isn't recognized properly and you can't operate it as intended.
Literal
Pen [obj-を] move distance [subj-が] short-[or-たり], speed [subj-が] slow-[or-たり] do [if-と], well gesture [subj-が] recognize-[passive-されず] think-way [gen-の] operation [subj-が] cannot-do.
~たり~たりする is the list-of-examples construction for actions or states: 'things like X-ing, Y-ing, and so on.' Combined with と (conditional), the sentence reads 'when you do things like X or Y, then...'. The classical negative ず after a verb chains clauses in a stiff/written style, equivalent to plain ~なくて. 思った通り '(just) as expected / as you thought' is a set phrase using 通り 'way / manner,' derived from 通る 'to pass (along).'