Consistency is the first thing to go if you don’t maintain your swing path – i.e. proper swing-plane – and/or lose your rhythm.
To stay in plane, make sure you initiate the stroke with your shoulders – the engine of the stroke – rocking them freely about your spine, with the arms, wrists and putter following as a single unit. Try to avoid thinking about taking the putter back with your arms or hands. To help recognize a good swing path attach a Frog Stick on the putter shaft while locking-in a good natural swing plane.
To maintain good rhythm, make sure have a very light grip pressure. A tight grip will allow you to manipulate the stroke rather than letting it happen naturally.
If you want to reduce all the sources of error in the putter, consider trying a Frankly Frog a putter you will love.