Catcher visits can be a bit problem. Usually not, but some are notorious for mound visits, like the NY Yankees. I was watching a game on the MLB channel featuring the other team's announcers and they, pretty aggressively, joked about it.
I'd say limit to once per inning, and if a time is called for after, umpire says "no time". Play ball. Might help some games some times.