I missed a whole year of schooling (medical issue). I chose to stay in my year and do the syllabus off my own back by lots of reading when I was healthy enough to do so, and a copy of what the syllabus was meant to teach me.
I stayed in the same groupings I'd previously had (good at most stuff, terrible at Geography and History simply because I have a terrible memory for dates and location names; could still give precise sequences and find root causes, but that's not what makes those studies at that level).
I'm all for giving extra tuition for those that need and want it. I'm also for streaming as it keeps a generalised level of achievement (and these days, it's simpler than ever to extend your own learning online). The perfect model would be one teacher, one student, or very few students, as that way you get the best out of each small group. However, resources aren't there to do that, so you get big classes, where nobody get the most out of their education.
The reason I hate the one size fits all, is because it's been done before and it's terrible. My primary school had that policy. I read Lord of the Rings when I was 5, and at school was forced to read the children's beginner books. I couldn't run through them at my own pace, I had to choose a book for the hour or so (which I read in about 2 minutes) then sit there bored. No advancement beyond the pace that the slower members of the class could achieve.
Same with maths. I picked that up early, but was never allowed to progress at any pace because it wasn't the pace the class as an entity moved at. Now, it wasn't as easy to practice and learn math back in the '70s as it is now, and certainly not as easy as it was to practice reading. So this ended up with my math ability to be stunted and held back, to the point that it damaged my scholarship chances when I went to secondary schools.
I was bored out of my mind by things I'd progressed well beyond, and _wanted_ to be challenged, but the school and teachers simply wouldn't let me.
It took far more effort than I'd have otherwise needed to expend to catch up with people who _had_ been allowed to be challenged (and I still think I'd be better than I am if I'd learned earlier, rather than having to fight to stay with the groups that'd been challenged to excel earlier).
That's direct experience. Now, I work in a field that does use math a lot (it's pretty foundational) and to a reasonably high level. All the ones that struggled and were the bulk of the class do not work in fields that need anything other than basic arithmetic. People advance to their strengths as they grow.
The counterpoint to this is that I love music and thoroughly enjoy playing. I'm a competent musician; however, no matter how much I practice or push, I'll _never_ be as good as the people who have a real talent for it. Same with a horde of other fields. I may grasp the basics and essentials, and enough to get by, but that's where my end point over the course of education should point to and be projected to cover. Aiming to improve the overall average is essentially saying "We aim for everyone to be mediocre", which to me is insanity. Challenge people. Get them to be as good as they can be where they can be. If they're challenged by the basics, that's not necessarily bad; they'll just be likely to only grasp the basics for quite a long time. If they're a prodigy, they should be taken as far as teaching can take them. That way, they may have a better chance of actually improving the overall system in their time, allowing everyone to eventually benefit from their excellence, rather than have them mired in the mud, and nothing progress unless it's at the rate that the average person could progress it (that actually leads to retreat from knowledge, as once things are at the pace of the average, what it takes the exceptional to understand today will be forever beyond their reach, and once that falls, exceptional is reduced and so on).