It's not what I say about size of a sport, it's how they compete. Many of those smaller sports don't compete in the usual NCAA divisions. Take skiing; wikipedia lists 29 NCAA programs TOTAL (https://en.wikipedia.org/wiki/...NCAA_skiing_programs). They compete regionally, not by Division 1-2-3. Same for fencing: standings are a mix of D1 and D3 (https://usfca.org/index.php/standings-men). Squash and archery are not NCAA-recognized sports at all, and schools from all levels compete together. Switching to D3 would not affect competition in these sports at all.
Ivy parents and players on this board always praise the schools for allowing students to take hard majors, etc. In baseball, Ivies play a 40-game season, like D3s. No-one goes to an Ivy to win the CWS, but they do go to win the Ivy League, and for the education. Yes, it's fun when Columbia knocks some team out of the NCAA tournament, but would it not also be fun if Columbia won the D3 CWS? (but would they?)
I'd note that the "country-club" sports are coming under increasing fire at schools like Ivies. Currently all an Ivy can offer is admission, and I wonder how long that will last? If squash, fencing, archery, rowing, and skiing were gone at those schools, the Ivy Director's Cup standing would collapse.