This is the US we are talking about, I think the reason is dumber than you think. The universities that are gaining are also in the big name athletic conferences. It's brand recognition
There are some other reasons. Affordability, more going to trade schools, some disillusionment with higher education. Honestly, I believe the biggest factor is kids want the big name school affiliation and if that doesn't happen, they either settle for a smaller school or go a different route.