Both are very difficult and I'm sure parents get a little nervous in either case.
It only bothered me if hitter son got himself out, like swinging at a bad pitch. Anything else was OK, I knew he would hit the next time up. If he got a hit I felt good.
I watched pitcher son set the all time record in the Pacific Coast League with 7 (yes 7) walks in one inning before they mercifully took him out. I doubt there has ever been a hitter that has had an at bat to compare with that. In fact, even a game as bad as that. If he struck out a hitter, there was no time to relax or enjoy. It's just one out and up steps a different hitter.
Lets say a pitcher throws 5 innings. Any given hitter might have two ABs and a couple plays to make on defense during that 5 innings. That pitcher is playing with success or failure every pitch he throws for those 5 innings. Even if he throws 5 perfect innings he has pitched to 15 hitters.
If the hitter has two chances to succeed or fail. The pitcher had fifteen chances to succeed or fail and more than that if he isn't successful getting every hitter out. If you get nervous watching your son hit or pitch, which one would create the most chances to be nervous?