It depends on how you approach your relationship with your son and the game. If you learn to just kick back and watch the kids play its fun. If you decide that your going to be stressed about every mistake your son makes or spend time trying to read into what the coach is thinking then it will no longer be fun. Heck it might even cause relationship problems with your son.
I was at my kids game yesterday. Sat with my wife for the first part of the game. She headed home early, once my son was removed from the game, to cook dinner. At that point I walked over by the "dads on the fence" to say hi to a few folks. It always interesting over there. Some dads are totally laid back, others fret over every pitch and every move the coach makes. To me it does not look like some of these dads are having fun.