It's tough. Everyone's situation is different. I can't imagine you ever regretting taking time away from coaching to spend more time with your own family. However, if you are like me, the kids that you coach are pretty darn close to family too.
Also, what is your work schedule? Coaching (baseball) definitely made it more complicated to go watch son's college ball but a regular work schedule also factors in, as does budget if you are talking flights and overnight stays. Depending on the year and the school, we were able to make a fair amount (3-5 trips /yr JC, maybe ten to in-state D2 and then only one for his senior year across the country after a transfer). I would liked to have made more games/trips but it was plenty to stay engaged with his experiences. In some ways, I think keeping it sporadic was good for him in terms of becoming fully independent and his own man more quickly. By the time he was a senior, he was totally aware that several cross-country trips weren't going to be feasible.
I would think that coaching football plays pretty well into being available for much of the college baseball schedule. And many HS's are done with their season by the time the fall series are taking place. Maybe you can just arrange to be away from football for part of that series. If you're HC, it's tougher. Maybe catching a few days between football is an option? Otherwise, if your son is anything like mine, he would totally understand if your FB team was making a playoff run. The fall series is, after all, not real games.