College coaches certainly attend high school games, but usually only on Mondays and only locally. If you live in an area close to several schools (say somewhere like Houston) high school ball can certainly be relevant. However, it's very limited.
As to using the summer to develop physically and not caring about decreased performance, for some it may be the opposite. It may make more sense to take that attitude with the high school season. That, however, would have its own problems - especially with teammates. The truth is that the vast majority of serious recruiting is done from mid-June to November. With year-round showcase opportunities, this gives more rather than less flexibility to build in rest periods.