NC Dad,
Yes, the umpire should make the pitcher change or remove the undershirt. Rules specify that sleeves of each individual player shall be approximately the same length and shall not be ragged, frayed or slit. If the pitchers sleeves are exposed, they shall not be white or gray. Also; a pitcher shall not wear any items on his hands, wrists or arms which may be distracting to the batter.