Perceptions are a funny thing. Everyone has a point of view and in the end they are all meaningless.
Some kids are small, others are big. However, baseball is a game which requires Heart, and I have yet to see a measurement for the size of one's Heart.
My son was never small, he blew through 6'2" and 190lbs at age 12. He was always big and squishy. Coaches never thought of him as athletic enough. There was always a kid that was a better "athlete."
Yet in the end you stick with the game, work on your weaknesses, grow mentally tougher and keep playing.
I have heard countless coaches say my son was too slow, it was why he wasn't recruited out of HS. Yet this season he led the NCAA in 2b's earning all-american honors, not bad for a big kid that is "slow."
In the end, it is the size of the Heart. Don't take no for an answer, keep playing, find a place where you are embraced for what you can do, not what you aren't.