I think it is too cold when the temperature affects the athletes' concentration. Generally for me that would be about 45 degrees. We have played in lower temps, but it sucks. Especially for baseball which is a less active game. Hard to stay warm.
As for the winter Olympic athletes, they have the latest and greatest technology to keep them warm. And I am certain riding down the luge track at 80 mph gets your adrenaline running so high you wont fell the wind chill.
Concentration is about mental toughness. My son likes playing when its colder or hotter than normal. He said let the competition focus on moaning about the weather. He'll stay focused on getting the job done. The only time he balked about the weather was when his high school coach wanted to move him from position to mound without warmup on a 20 degree wind chill day. He asked to come out of the game for one hitter to get in some warmup tosses.
While I agree with you about mental toughness, there is a reason the professional teams work out in a warm weather climate. There is also a reason that most northern football teams have indoor facilities, even though they will play outside on Sunday. I wouldn't tell them they aren't mentally tough. 
Coach2709 has the right approach. Start out indoors with the selective use of outdoor cold weather situations when necessary.