Here's what I do:
If a middle infielder or center fielder is clearly calling for the ball and backs away at the last second, I give the error to them. By the rights of baseball, they have the ability to call everyone else off; if they do, they get the error if everyone backs away.
Likewise if an infielder calls a catcher or pitcher off a pop-up and backs away, they get the error. If everyone just kind of looks at each other, the fielder nearest the ball gets the error.