Comment Re:Duh (Score 4, Informative) 233
Actually this is pretty widely studied at this point. It is one of the reasons that AI companies have pushed for marking AI generated content. At the most basic level LLMs are probability models where they predict the next most probable word given what has come before. If you think of it like a gaussian peak they are trying to select near the peak. If you train an LLM on the output of an LLM you are training it on just the peaks and you get a kind of focusing effect after only a couple generations and the model collapses.
It turns out you need the low probability information to keep the model stable and it can't generate that information.