Given that an operating system consists mostly of a bunch of drivers attached together with a kernel, there are good reasons to prevent distribution of closed drivers mixed with GPL ones. I don't think it is legal, not without stretching the meaning of the GPL.
Consider the following scenario:
Intel develops new closed undocumented architecture with a 16 core cpu. Similarly to current network or video cards, you need a proprietary driver to enable the super accelerated multicoreness. In order to allow the use of the newer faster cpu's, Linux vendors do what they did with the other proprietary drivers, label these drivers as "not part of the kernel" put them in a wrapper and ship their version of Linux with the proprietary drivers which, for now, intel is giving away for free as a binary blob. For a while everybody is happy and content. The new 16 cores chips becomes the norm. There are even 32 core chips on the market and the 64 cores chips are soon to be released all of which rely on proprietary drivers.
Suddenly, we hear that a large company, Lintelsoft, started by ex MS executives, makes a deal with Intel, a very lucrative deal for Intel, to license the drivers. Intel then says they won't give away the drivers anymore but you are free to buy the brand new Lintel Linux distribution. This distribution, which sells for 699$ a piece is all GPL'd except for those drivers that have become so prevalent that you need them in order for computers to run at a reasonable speed.
Open source programmers scramble to write free replacement drivers that work on their Gnubian distribution but only manage to make drivers that can run the multi core cpu's at 1/20th the speed as Intel won't release documentation or specifications. Linux is rendered mostly useless except for the Lintel distro, (which is also available for free and with sourcecode as Lintelora, excluding the proprietary driver sources of course) You can always plug in the Gnubian drivers in the free Lintelora project and get a working computer but it will only run at 1/20th the speed of the commercial 699$ a pop version and isn't powerful enough to run the new Mozilaurus browser smoothly.
In this scenario, Lintelsoft would have effectively stolen Linux from the open source community, making profit with other people's source code and breaking all versions that are free.
How can we let anyone close up an obviously derived work based on some wrappers?
Notice that, even today I sometimes need to pay to get a fully working Linux from certain vendors, like Mandriva. (if i don't pay, 3d acceleration wont work.) I expect that kind of twisting of the law by commercial vendors. It surprises me that even Ubuntu is including proprietary video drivers nowadays.
What's worst is that legally in order to maintain copyrights you need to make reasonable efforts at protecting those rights. Legally if the open source community waits until the binary drivers become problematic before acting, proprietary vendors will be able to argue legitimately that closed source code has been allowed in the kernel by the open source community for a long time now: The law says that you are not legally allowed to suddenly change your mind about interpretations to suit current needs thus the open source community would be screwed.