The pitcher can still step off, even if in the full windup position. Once he steps off, he can do anything he likes to try to put the runner out. To do this properly, his very first motion must be to lift the pivot foot and step backwards.
BUT, once he starts his pitching windup motion, he has to go home or else it's a balk. So in the question as put, it's a balk because as I understand it he had already begun the windup, with hands going overhead before he turned to go to first base.
Typically the first pitching motion will be when the kick leg foot is picked up. Once that happens, it's go to home, or balk.