When the OP says his son is "very smart," I think about my son, who has a late July birthday and is the youngest in his 2011 class. I remember a meeting with his second grade teacher. We were concerned because he would cry the whole time when he had to do his reading homework because they had already read the story in class. She was a wonderful teacher. Ms. Crawford. Smiling the entire time, she told us that he and his buddy cut up all day long, but when she was trying to teach, and would ask 2B "what did I just say," he would repeat it verbatim. She separated him from his buddy and let him read Goosebumps books when he was done with his assignments, and everything was fine. And she said it was fine if he read "SI for Kids" at home instead of the textbook.
For him, I think it would have hurt him to hold him back - he may have been bored, gotten in more serious trouble, and taken an entirely different path. One never knows, but 2B is exactly where he needs to be, and I have no regrets.