I used to co-coach a travel team (6 years) and we saw this problem a lot. I definitely agree with DadOfPlayer and respectfully disagree with TRHit. (Wondering if TR's experience is above the youth travel level, where these problems are generally not present.)
As with the case of the overbearing or screaming coach, it is absolutely your responsibility as parent to remove your child from a team that is not led by an adult who is setting a good example of what an adult leader should be. This is not "quitting". A lot is asked of the kids and the families on these teams and the "loyalty to the team" thing has got to work both ways or it doesn't work at all.
At ages 12-13, there are probably 3 travel players for every 1 high school varsity slot those kids will compete for in 4-5 years -- at least in our area. The better players start to leave their local teams and gravitate to more elite teams where they all play together. The teams left behind struggle to survive and eventually, they don't. One of the big signs that you're on the wrong kind of team is when you have a coach who makes it the mission of the team to serve his one son. This guy is not really going to help your son make the cut down the road, and he is (irrespective of what any one family does) going to run off other families to the point that you are destined not to be one of the better teams out there. One day you won't have enough kids to field a team and that will be the end of that.
Being a player in this situation is kind of like being an employee. If you become a "team hopper" (like a "job hopper"), you'll get a reputation for being someone who just is never satisfied. But leaving a team once, or just once every few years, to find a better situation is common and will not hurt your son. In fact, most of the other teams out there probably know all too well what's going on on your team, if only from playing against you over the years. If you handle this with class, people will respect what you're doing.
You probably also know who the class acts are among coaches, from watching those other teams you've played over the years. Call those teams first, and you'll avoid having to jump teams more than once.
BTW, one way we used to try to avoid this problem was by keeping our team roster at 11 players. Lots of playing time for everybody! A coach with a son on the team can probably never completely escape negative thoughts from other team parents, but if you're genuinely fair, you can keep the carping in the stands to a minimum.