For me a lot depends on the batter. If the pitcher wants to work fast (maybe even very fast), and the batter is ready for the pitch - meaning he's where he wants to be in the box and is looking at the pitcher, I'm all for it.
If the batter is not ready, you must prevent the pitch. I will call time and bark at the pitcher to wait until the batter is looking at him before he starts his motion to pitch.