The issue isn't to which pitcher the conference is charged. It's who's at bat. Both OBR and NCAA forbid multiple conferences during the same batter.
NCAA specifically says that the at-bat starts at the moment the previous batter reaches base or is put out. Thus, no second visit allowed.
OBR is vague as to when it starts. Given that the wording on the restrictions is identical, I'm not allowing this...unless there's an interpretation out there that says it is.
This is from MLBUM:
If a coach goes to the mound and removes a pitcher and then the manager goes to the mound to talk with the new pitcher, that will constitute one trip to that new pitcher that inning.
If a manager changes pitchers and leaves the mound, he (or a coach) may come out again to visit the pitcher while the same batter is at bat, but this will constitute one trip to that new pitcher that inning.