Today I watched the Univ of Richmond and Bucknell game with my youngest son. It was 35 degrees with a 10mph wind. There was one hit into the 4th inning. The hitters didn't look comfortable. The pitchers were "dealing" and thankfully moving the game along. Both pitchers were wearing sleeves, and were allowed to blow on their hands.
I've seen a bunch of cold weather games in the last few years. I've seen games played in snow and freezing rain that turned into pitchers duels. I think the pitchers and catchers are fine because they are constantly moving. I think the position players and hitters have a more difficult time. A pitcher can much more easily own that inside half of the plate, and make that hitter just a little bit uncomfortable in the colder weather.