Here is why it matters:
1. The top boarding schools have reputations, connections, relationships with college admissions, unbelievable resources given high endowments (larger than some colleges), and the best and brightest from around the world. Independent schools have most of this too, but not the best and brightest from around the world as they are city-centric or somewhat regional.
2. If you look at JUST athletes and those who move on to play in college (all sports), then you will find that the elite prep schools have more kids who have repeated a grade. This is because (a) they take PGs so there is that immediate difference and (b) most independent schools start earlier than 9th grade and therefore some of the kids who go there who are also athletic do not repeat. Yes some still repeat, but not to the extent of the elite prep schools.
3. Not all privates disclose ACT/SAT scores. Judging anything off of such scores is probably not the best way to evaluate a school. This may also be skewed by the school's admissions dept taking high scoring kids on the SSAT etc in the first place. You are right about Roxbury Latin, it is an awesome school.
When looking at a private, boarding or otherwise, more relevant to look at the college matriculation list over the past 5 years, the number of faculty with advanced degrees, the student teacher ratio, the number of classes and depth of their offerings (hey one of the elite prep schools offers a English class all about BASEBALL!!), the length of your commute if a day student (with school, sports and the needed amount of sleep, the commuting time can be a real detriment), FA if an issue, the facilities (ie if a hockey player, does the school have one rink or two?), etc. Records and coaches change.
4. There is a trend in elite college admissions to increase the number of public school admits. This means the privates are taking a hit. Not true for under represented minorities at privates, not true for recruited athletes obviously, and not true for development candidates ($$$$$ donated). But it is harder for non hooked private school kids in the past 5 years. This is only trending downward. Having said that, the very top elite boarding schools are not getting hit to the extent of the independent schools, but everywhere is trending downward.
No one is saying you have to go to XX (name brand well known) school just because it is well known. That translates to going to Harvard just because its Harvard. IT IS ALL ABOUT FIT. Whether HS or college, this is the most crucial point.