I would agree with what @Dominik85 shared. Sports like running/swimming have concrete times. You're either fast enough or your not. A sport like tennis is also relatively easy for evaluating ability. Basically have two people play a set and you'll determine who's going to win the majority of the time. Baseball...that's a multi-person team sport, so it's hard to figure out how much of an impact a singular player can have on the team's overall performance.
My family had a discussion the other day about what we called "equalizer" sports. Sports that don't provide large advantages due to physical shape. For example, we all decided basketball is a sport where having height gives quite a huge advantage. Wrestling was one we decided, due to weight classes, that sort of equalized competition. We felt that swimming was one that definitely gave advantages to those with height. Football was another, that's probably why sprint league has come about. Baseball, was a mixed one. As a pitcher, we felt size made a huge difference, but for a position player we didn't feel that being big gave a huge advantage. Sure a bigger person will probably generate more power when hitting/throwing, but with hitting we felt coordination is what made the biggest impact.
Obviously, when one trains for a specific sport though, the body physically changes to better perform in that sport. I know for myself I would swing 20lbs in weight as a petite female each year. I was a lightweight during XC and in the spring I was at my heaviest as a sprinter. I would never get any taller, but between being a distance runner or a sprinter...my genetics were better programmed for sprinting.