When our son was going on unofficial visits, we would stay out. His visits were his visits. On one occasion, at a PAC 12 school, the HC asked him if he was alone, he said his parents were somewhere around. HC said go get them & bring them back to his office. He visited with us for about an hour & ended up showing us the facilities. That was pretty cool.
The only other times we accompanied him was on official visits.
Not sure what the right answer is for you. We just felt it was journey. In hindsight, there are many things we might have done differently, just not sure that was one of them.