It's really hard to gauge things from a post, but it seems like you might be putting a lot of pressure on your son. I think it's one thing for a kid to lead the charge on what's important, but gong through every scenario in your head and stressing about it, will likely be putting a ton of pressure on him whether you verbalize it to him or not. I don't know his (or your) ultimate goals but there is a ton of time to be stressed out over life, if you can pull some of that back when they are still kids, I'd do that. I'm not saying that you don't have accountability or expectations, I just wouldn't steer the ship unless it's what he is wanting. (I apologize if I'm way off base).
If his goal is IVY, then maybe just sit with him and make a chart of pros and cons and let him decide what is right for him. If it's not IVY or HD3, then I wouldn't stress over it at all.