I think it is too cold when the temperature affects the athletes' concentration. Generally for me that would be about 45 degrees. We have played in lower temps, but it sucks. Especially for baseball which is a less active game. Hard to stay warm.
As for the winter Olympic athletes, they have the latest and greatest technology to keep them warm. And I am certain riding down the luge track at 80 mph gets your adrenaline running so high you wont fell the wind chill.
Concentration is about mental toughness. My son likes playing when its colder or hotter than normal. He said let the competition focus on moaning about the weather. He'll stay focused on getting the job done. The only time he balked about the weather was when his high school coach wanted to move him from position to mound without warmup on a 20 degree wind chill day. He asked to come out of the game for one hitter to get in some warmup tosses.