This is interesting as mostly men have responded Lefty34. If you don't mind, I would like a crack at this one. I don't believe it is the parent's job to worry about or manage their son's girlfriends. It is their own personal business imho. When my son left for college, he was dating a high school junior and my wife and I were very fond of her. When it was her turn to pick a school, she chose my son's college 750 miles away. My wife and I feared that was a mistake because it is a small campus, small town, and we thought they just might see too much of each other. It turns out we were right as they ended their realtionship last year after many years of dating. We believe our son may have been the controlling one in that relationship and we believe he may have learned a valuable lesson the HARD way. We were very sad that things did not work out but I still believe it is their life and they need to learn these things on their own.
Enough pontificating however... I'll try and answer your question....
What if your son were unattractive to girls? How do you suppose you would feel then? How do you suppose he would feel? IMHO, the way to handle your feelings on the present matter is to see it for the blessing it is
Someone else see's the special qualities in your son that you do
Moreover, your son is plenty old enough to figure out how to handle a rude/intrusive friend (male or female) on his own imho. I hope I was able to help in some small way and apologize in advance for being frank and posting in this forum quite frankly.