It's because everyone else has other responsibilities on the play, and have (at best) nothing to offer. At worst, they have incorrect information based on them being in a bad position to see what happened. There are absolutely times to get together...MC at the plate is not one of them.
In short, going to someone else on plays that are not their responsibility is more often to lead to an incorrect call than a correct one. MLB figured that out in very short order, which is why replay was accelerated. The camel put its nose in the tent when they "encouraged" crews to consult on disputed plays...then they found out that it was having the opposite outcome than what was desired. Since they couldn't go back to the way it was, replay got significant impetus.