as the others have said, pitchers have it easier...although, it can't be comfortable...and I'm sure the coaches limit their pitch counts more in cold weather (as a club coach, I know I do).
But it is far worse for a position player than the pitcher. Worst game I ever played in was in N. Texas in early April... crazy cold front came through after 1st inning of a 7:00pm game... wind out of the north at 25-30 and temps plummeted to around 35 within 30 mins. I think it was bad b/c we did not dress prepared for it, but that was our fault as dumb college guys. I will never forget trying to throw a ball across the diamond from SS... more like a shot put b/c my fingers were numb. Anyway, the games in cold weather move quickly, as long as the P is willing to pound the zone and not nibble (although the umps are cold, so likely the zone gets a little wider, too)... batters take hacks early in the count.