These visits can be very casual. I don't know that there's a lot to worry about as long as you avoid the obvious "don'ts."
As in, don't wear a cap that still has a sticker on it, don't wear your cap sideways, don't wear your pants around your knees, don't go showing everyone your boxers, etc.
I've seen guys wear a decent tee and shorts and be just fine. Me, I'd go with khakis and a polo just to make sure the coach perceives that you understand the need to show respect.
It may be a job interview, but it's not a coat & tie situation at all.