under Fed rules taking the sign from off the rubber is illegal and is administered as a balk if there are runners on base.
As an aside, I have never called this as a balk. This is a "just don't let it happen" situation. If he looks in for a sign from off the rubber just call time and tell him he must be on the rubber to take the sign. If you are at the plate then tell your catcher not to give the sign if he isn't on the rubber. Once you explain the consequences he will help you out with that.