Agreed. The summer in between is still an important time despite already being committed to a school. The last thing you want to do is decrease the talent that your son is facing or competing against. You don't want to go in to school having faced 78-82 mph pitching for 2 months over the summer if you are more likely to see 88-92 when he steps on the field for the fall.
Playing with former or current college guys couldn't hurt as long as the competition is still good.