OpenAI's Next Big AI Effort GPT-5 is Behind Schedule and Crazy Expensive (msn.com) 120
"From the moment GPT-4 came out in March 2023, OpenAI has been working on GPT-5..." reports the Wall Street Journal. [Alternate URL here.] But "OpenAI's new artificial-intelligence project is behind schedule and running up huge bills. It isn't clear when — or if — it'll work."
"There may not be enough data in the world to make it smart enough." OpenAI's closest partner and largest investor, Microsoft, had expected to see the new model around mid-2024, say people with knowledge of the matter. OpenAI has conducted at least two large training runs, each of which entails months of crunching huge amounts of data, with the goal of making Orion smarter. Each time, new problems arose and the software fell short of the results researchers were hoping for, people close to the project say... [And each one costs around half a billion dollars in computing costs.]
The $157 billion valuation investors gave OpenAI in October is premised in large part on [CEO Sam] Altman's prediction that GPT-5 will represent a "significant leap forward" in all kinds of subjects and tasks.... It's up to company executives to decide whether the model is smart enough to be called GPT-5 based in large part on gut feelings or, as many technologists say, "vibes."
So far, the vibes are off...
OpenAI wants to use its new model to generate high-quality synthetic data for training, according to the article. But OpenAI's researchers also "concluded they needed more diverse, high-quality data," according to the article, since "The public internet didn't have enough, they felt." OpenAI's solution was to create data from scratch. It is hiring people to write fresh software code or solve math problems for Orion to learn from. [And also theoretical physics experts] The workers, some of whom are software engineers and mathematicians, also share explanations for their work with Orion... Having people explain their thinking deepens the value of the newly created data. It's more language for the LLM to absorb; it's also a map for how the model might solve similar problems in the future... The process is painfully slow. GPT-4 was trained on an estimated 13 trillion tokens. A thousand people writing 5,000 words a day would take months to produce a billion tokens.
OpenAI's already-difficult task has been complicated by internal turmoil and near-constant attempts by rivals to poach its top researchers, sometimes by offering them millions of dollars... More than two dozen key executives, researchers and longtime employees have left OpenAI this year, including co-founder and Chief Scientist Ilya Sutskever and Chief Technology Officer Mira Murati. This past Thursday, Alec Radford, a widely admired researcher who served as lead author on several of OpenAI's scientific papers, announced his departure after about eight years at the company...
OpenAI isn't the only company worrying that progress has hit a wall. Across the industry, a debate is raging over whether improvement in AIs is starting to plateau. Sutskever, who recently co-founded a new AI firm called Safe Superintelligence or SSI, declared at a recent AI conference that the age of maximum data is over. "Data is not growing because we have but one internet," he told a crowd of researchers, policy experts and scientists. "You can even go as far as to say that data is the fossil fuel of AI."
And that fuel was starting to run out.
"There may not be enough data in the world to make it smart enough." OpenAI's closest partner and largest investor, Microsoft, had expected to see the new model around mid-2024, say people with knowledge of the matter. OpenAI has conducted at least two large training runs, each of which entails months of crunching huge amounts of data, with the goal of making Orion smarter. Each time, new problems arose and the software fell short of the results researchers were hoping for, people close to the project say... [And each one costs around half a billion dollars in computing costs.]
The $157 billion valuation investors gave OpenAI in October is premised in large part on [CEO Sam] Altman's prediction that GPT-5 will represent a "significant leap forward" in all kinds of subjects and tasks.... It's up to company executives to decide whether the model is smart enough to be called GPT-5 based in large part on gut feelings or, as many technologists say, "vibes."
So far, the vibes are off...
OpenAI wants to use its new model to generate high-quality synthetic data for training, according to the article. But OpenAI's researchers also "concluded they needed more diverse, high-quality data," according to the article, since "The public internet didn't have enough, they felt." OpenAI's solution was to create data from scratch. It is hiring people to write fresh software code or solve math problems for Orion to learn from. [And also theoretical physics experts] The workers, some of whom are software engineers and mathematicians, also share explanations for their work with Orion... Having people explain their thinking deepens the value of the newly created data. It's more language for the LLM to absorb; it's also a map for how the model might solve similar problems in the future... The process is painfully slow. GPT-4 was trained on an estimated 13 trillion tokens. A thousand people writing 5,000 words a day would take months to produce a billion tokens.
OpenAI's already-difficult task has been complicated by internal turmoil and near-constant attempts by rivals to poach its top researchers, sometimes by offering them millions of dollars... More than two dozen key executives, researchers and longtime employees have left OpenAI this year, including co-founder and Chief Scientist Ilya Sutskever and Chief Technology Officer Mira Murati. This past Thursday, Alec Radford, a widely admired researcher who served as lead author on several of OpenAI's scientific papers, announced his departure after about eight years at the company...
OpenAI isn't the only company worrying that progress has hit a wall. Across the industry, a debate is raging over whether improvement in AIs is starting to plateau. Sutskever, who recently co-founded a new AI firm called Safe Superintelligence or SSI, declared at a recent AI conference that the age of maximum data is over. "Data is not growing because we have but one internet," he told a crowd of researchers, policy experts and scientists. "You can even go as far as to say that data is the fossil fuel of AI."
And that fuel was starting to run out.
That's not how humans learn (Score:3)
On the other hand, if someone wants to pay to maximize the potential of this algorithm, great! It's not my money, and I'll get to see what it can do.
Re: (Score:2)
Why are we even trying to mimic humans? Humans are pretty stupid and can be guided into even greater stupidity and evil.
Re: (Score:3)
Re: (Score:3)
Humans are vastly more efficient in terms of how many "tokens" they require to learn the same amount.
Humans learn by moving around, observing, and interacting with the world.
The human eye can process 30 frames per second at over 100-megapixel resolution.
By the time a two-year-old learns to talk, they've processed about 200 quadrillion bytes of data.
Re:That's not how humans learn (Score:5, Interesting)
By the time a two-year-old learns to talk, they've processed about 200 quadrillion bytes of data.
Even if that's the case, and even if all that data is actually used for brain remodeling, how much of it isn't redundant? According to the predictive brain theory (which is not proven), data gets dropped at every level of brain processing to the extent that it matches expectations of the deeper neural layers. Only the errors get propagated, which may be substantial for a two year old, but means for example that once objects can be recognized, very little visual data about any known object needs to be processed much beyond the optic nerve.
Re: (Score:3)
Also, a lot of knowledge and insight does not require all that real world experience at all.
Mathematics is incredibly pure a priori knowledge, so even without any data about the universe an ANN should theoretically be able to reach perfect scores on every challenge. The (lack of) data is clearly not the problem here.
What we need is further architectural and/or training process improvements. "Prospective configuration" (alternative to backprop) might be interesting for the latter.
Re: (Score:3)
There's a saying: "organisms organise".
Putting aside evolution/design, we seem to arrive as organisms ready to model and interact with the world in certain ways; the life of a dog, the life of a human, the life of a bacterium.
I for one am stunned by what LLMs can do -- but is it maybe fair to say that they are very high powered and large scale guesswork?
Is it more fair to say that they're very focussed powerful pattern recognition? And hence incredibly useful for certain specialised tasks. Maybe they can s
Re: (Score:2)
Putting aside evolution/design, we seem to arrive as organisms ready to model and interact with the world in certain ways; the life of a dog, the life of a human, the life of a bacterium.
You can't put aside evolution here. It has led to the topological design of our neural networks and of course that is tailored to interact with the world in certain ways (and do so very efficiently).
For the stuff that organisms need to do to survive on this planet the effective amount of data, amount of training and topology refining that has been done over billions of years is enormous. Now for ANNs, we are feeding them shitloads of data and training them very very much. The evolution of ANN topology howev
Re: (Score:2)
Yeah, nobody does mathematics well without quite a lot of existing experience. I'm not even sure what you mean by "mathematics is incredibly pure a priori knowledge." Mathematics is based on a set of assumptions that have taken several thouand years to get mostly, we think, right, and all of the "knowledge" represented by pure mathematics is implied by those axioms.
I suppose you could argue that once you have those the rest requires no additional contact with the real world, which is technically true. A per
Re: (Score:2)
I'm not even sure what you mean by "mathematics is incredibly pure a priori knowledge."
See https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2F... [wikipedia.org]
But choosing theorems that are interesting, and pruning the search tree to practically prove specific theorems, requires an immense amount of experience.
Experience in math, not with the universe. Unless you are specifically referring to applied mathematics.
Please don't miss my point by getting bogged down in details. Even if math required some real world experience, you must be able to see that there are huge differences in how much real world experience/data is needed for understanding and being able to effectively reason in different disciplines.
Re: (Score:2)
I was giving you the benefit of the doubt. If you click on the "Immanuel Kant" link in your Hume link, then on the "Critique of Pure Reason" link, you will read about how Kant made all kinds of friends by pointing out that considering mathematics specifically as a priori knowledge is fukken weird. In his words: "it is a "matter of life and death" to metaphysics and to human reason, Kant argues, that the grounds of this kind of knowledge be explained."
The
Re: (Score:2)
The human eye can process 30 frames per second at over 100-megapixel resolution.
Citation for that claim please.
Re: (Score:2)
The human eye can process 30 frames per second at over 100-megapixel resolution.
By the time a two-year-old learns to talk, they've processed about 200 quadrillion bytes of data.
This is way off. The resolution of the human eye is closer to 1MP and the total bandwidth of the optic nerve is on the order of 10mbit/s.
People sometimes make breathtaking claims with regards to human vision by discarding the fact eyeballs move in their sockets and have an insanely narrow field of view with any detail. Don't move your eyeballs and read what is a few lines up down or in any direction. Human vision is mostly illusionary.
Re: (Score:3)
Humans are vastly more efficient in terms of how many "tokens" they require to learn the same amount.
The human mind is a machine so “efficient” it’s born in completely helpless and totally stupid state, and requires a proverbial village just to raise it just to become capable enough to start learning. Provided you’re rich and lucky enough to be not born into poverty, since that human mind requires “tokens” in the form of food and every few days (minimum) to simply survive. If “tokens” were viewed as dollars, we’re probably talking $250-500K from birth
Re: (Score:2)
I've seen a new born, smart enough that it knows what it wants, a boob, and smart enough to find and suck on that boob.
Re: (Score:2)
Humans are vastly more efficient, at assuming.
And yet, we humans are harsh critics of AI hallucinations, even though the machine hallucinations pale in comparison to the quantity and quality of human hallucinations. Of course, we don't label human hallucinations as such. Rather we use words like assumptions, extrapolations, inference, confidence, and bravado. It's a good thing we don't use the same standards for hallucinations for our human friends because we likely would have no friends.
Re: (Score:2)
Humans are vastly more efficient, at assuming.
And yet, we humans are harsh critics of AI hallucinations, even though the machine hallucinations pale in comparison to the quantity and quality of human hallucinations. Of course, we don't label human hallucinations as such. Rather we use words like assumptions, extrapolations, inference, confidence, and bravado. It's a good thing we don't use the same standards for hallucinations for our human friends because we likely would have no friends.
If a person I call friend is acting a bit too assumptive with excessive confidence and/or bravado, I’d probably remind my friend of how delusional they’re acting. Fitting enough word to describe a hallucinatory mind, with the ability to both educate and offend depending on reception.
Re: (Score:2)
Re: (Score:2)
Humans don't need to read terabytes to learn how to read.
An AI can learn to translate text to speech in a few hours of training.
A human takes five years.
Re: That's not how humans learn (Score:2)
Using human language is one of the few things the current "AI"s can do decently.
Re: That's not how humans learn (Score:3)
And the result is a voice that's totally emotionless and not able to handle acronyms like Sgt., Lt. Etc.
Re: (Score:2)
Those aren't acronyms, maybe you need to go back to training.
Re: (Score:2)
Well, Sgt is spoken out as Sergeant, but the A.I. says "S.G.T." in a way that just sounds wrong.
Re: (Score:2)
An AI can learn to translate text to speech in a few hours of training.
Assuming that's true (I don't know what study you are citing), then how many gigabytes of text does it have in its training set? It's clear that the way humans learn is not the same as (current) neural networks.
Re: (Score:2)
That was clear from the very beginning. There was never any confusion about that. Neural networks are nothing like brains. If you believe otherwise, you're deeply confused. Why this is so difficult for the the Kuzweil acolytes and Lesswrong cultists to accept, I'll never know.
Re: (Score:2)
how many gigabytes of text does it have in its training set?
A decent text-to-speech engine needs about 100 hours of recorded and transcribed audio for one voice.
At 100 Kbps that's four gigabytes of audio.
The text is 0.1% of that, about four megabytes.
Re: (Score:2)
You're desperately trying to justify a nonsense belief. Why? What is it you want to be true about AI?
Re: That's not how humans learn (Score:2)
Re: That's not how humans learn (Score:2)
Re: That's not how humans learn (Score:2)
Re: (Score:2)
Kurzweil is a crackpot. Find better heroes.
Re: (Score:2)
He answered the question being asked. Look around. Seems like the right answer. WTF is an LLM?
The average response of what 1 million people would say. Can't think beyond it. Yep. Nailed it.
No one said he's a hero.
Re: (Score:2)
Not all of them. What they try is to emulate the about 10-15% "independent thinkers" (the term is basically a polite lie to make the unthinking masses feel better) that can actually understand things and can fact-check. But turns out, nobody knows how to make a machine do that. So they fake it and about 85-90% of the population cannot tell the difference.
Re: That's not how humans learn (Score:2)
It's not your money for now, but it will be your money that will be used to bail out several multi-billion dollar companies when the moment of reckoning comes, because their demise would tank the whole economy.
Re: (Score:3)
But in the
Re: (Score:2)
Of course there's a better way. Some tech entrepreneurs saw some language model demos, decided they had reached the point where they were "scalable" and did their thing. That thing involved some pretty ridiculous extrapolation, but that's what tech entrepreneurs do.
Complexity is a bitch (Score:4, Interesting)
The problem with algorithms is that you have to think of every likely scenario. "AI" provides a shortcut through real world examples training statistical models. Then whatever the given data adds up to as most likely becomes the solution to each new scenario. The problem is, this is just a hack, a brilliant hack, but just a hack. The belief is that, with more complexity in the model, more scenarios can be handled. That works quite well and amazingly, up to a point. Then the shit hits the fan, and the model can't cope. There are 10^40 in chess, and that shit's a lot easier to deal with than autonomous driving, which faces a lot more than 10^40 scenarios.
Re: (Score:2)
It isn't just complexity, the real world deals up Black Swan Events, i.e., events that have little to no precedent. Humans have difficulty dealing with them and they generate most of the training data. It wouldn't be surprising to see that AI similarly has difficulty.
Humans more or less work on the concepts of continuity, tangents to curves, and not recognizing that one only has partial and not complete information. They assume the world will work in the future as it has in the past. They also assume all th
Re: (Score:2)
It is even worse. A "Black Swan" for an LLM is a completely standard problem that has a tiny bit of uncommon border conditions that no smart human would ever have problems with.
For example, I recently gave students in an "open Internet, AI allowed" exam a standard algorithm to code, but I changed some steps in a way that does not make much sense but still works. 17/20 returned the standard algorithm the LLM (likely ChatGPT in most cases) gave them. 3/20 noticed the LLM answer was crap and fixed things or co
Re: (Score:2)
one thing we find here is not how smart LLMs are, but how not-smart many humans are
Agreed, and noting one sub-category of that not-smart-ness is trusting/assuming. I was describing to someone a difference between the AI used to mine data (or a number of other not-directly-consumer-facing tasks), and AI chatbots, the diff being that interaction with a chatbot has enough similarity with how we interact with other humans that it's easy for (non-alert) people to presume that this similarity extends far beyond its actual boundaries... and can even be hard
Re: (Score:2)
Exactly. If the humans are reasonably smart that is and notice the situation is non-standard.
Re: (Score:2)
Indeed. There is also a second problem: Statistical models cannot model complex things. They are always shallow. So complex things need to be in the training data and they need to be in there many times. And hence the training data blows up massively for a tiny improvement in the model. We probably have seen the maximum that is possible with LLMs within less than one order of magnitude because that much more training data is simply not available and cannot be created.
news companies dumping their gpt5 stuff over the w (Score:2)
OpenAI announces their gpt4 o3 reasoning model that meets some guys definition of agi and now editors are rushing to publish their "gpt5 when?" articles they had queued up in the hopper (and already paid journalists for) to run over the holidays, half baked no less.
looking forward to all the articles about the new o3 capabilities in January once journalists have a chance to wrap their head around how it works/what it does/how it's different
Re: news companies dumping their gpt5 stuff over t (Score:2)
Re: (Score:3)
Prepare to be disappointed. There's nothing mysterious here. The so-called "reasoning" models still can't reason. The "difference", if you can call it that, is that part of the output is hidden from the user. This is why responses take longer. The model isn't "thinking about the prompt", it's doing the same thing it always does: predicting the next token, one at a time. The basic idea is to get more (hopefully) related information in context for the output actually shown to the user. That's all.
If you'
Re: (Score:2)
Yep. Essentially, they are blowing more smoke and adding more mirrors to make the illusion more convincing.
Re: (Score:2)
It pears repeating that only utterly corrupted and dishonest "definitions" of AGI are met by any LLM in existence, and that will not change.
Re: (Score:2)
Interesting references. Thanks.
Gosh (Score:2)
Improvement in LLMs starting to plateau. (Score:2)
Re: Improvement in LLMs starting to plateau. (Score:2)
Re: (Score:2)
In certain circumstances, you can manufacture good synthetic data, certainly good enough that you can train a model on synthetic data.
However "certain circumstances" is doing a lot of heavy lifting.
I also disagree that LLMs are unusable for anything serious. I've definitely found them useful for what amounts to very fuzzy documentation search. Sure they hallucinate, but it's not like googling for stack overflow answers is 100% error free either. It's pretty neat that you can describe API functionality and i
Re: (Score:2)
a tool is only as good as the tool user
geez people, these LLMs are useful, people can write much faster using one, all people need to do is direct it and oversee it
stupid people don't seem to get intelligence
Re: (Score:2)
LLMs empowering you to write faster isn't what we're being promised.
We're being promised a machine that will 100% replace you and will make you useless.
We're being promised that you won't need salespeople, lawyer, and especially you won't need no stinking expensive "Developers" since our AGI will 100% replace devs.
The problem is that this isn't materializing. It's all promises and iterative steps on the same concept, the GPT. A writing machine. Now they tell you that this writing machine can reason, but it
Re: (Score:3)
They're not fooling anyone
Unfortunately, they're fooling quite a few people. Are you familiar with Gell-Mann amnesia? When you read a newspaper article about a subject you know well, you'll be horrified and all of the obvious errors and misunderstandings, but completely forget about that unreliability when you read articles about topics you don't know well. This is why people think chat bots are a good substitute for search. They get a confident answer that appears to answer their question and as long as it seems plausible, as i
Re:Improvement in LLMs starting to plateau. (Score:4, Interesting)
LLMs are not the same thing as AI
AI is a broad term that covers quite a few things you wouldn't consider AI. I'll point out that the science fiction version of AI came after the term was introduced at the Dartmouth conference. LLMs are AI. So are decision trees and linear regression.
but when this stupid bubble bursts it will drag the whole industry down into another 20 year AI winter.
Indeed. Do you remember the insane things people were predicting in early 2023? Total disruption of every industry, mass unemployment, a complete realignment of the social order ... by mid summer. Given how expensive they are to train and operate (less expensive doesn't mean inexpensive), it's amazing we haven't seen things crash already.
LLMs were always going to be a technological dead end. This is because there is literally nowhere for them to go - other than to try to scale up the models.
We're well-beyond the point where doubling the size of the model would net a noticeable improvement in performance. Performance gains tend to fall off logarithmically with scale. Transformers are no different. My expectation from the beginning was that the future of transformers would be in smaller, more specialized, models.
Getting them to do anything useful - like reasoning - has to be a bolt on.
They can't reason. What they call "reasoning" is just hiding the initial output from the user to hopefully put more relevant information in context for the output the user will see. There are "bolt on" features which call out to traditional algorithms, which helps to hide the very real limitations of these things.
hallucinations mean they can never be used for anything serious.
Transformer models are well-suited for language translation, speech recognition, and speech synthesis. As a lawyer, programmer, teacher, scientist, or customer support person, they're worse than useless.
the funniest thing here is the idea you can manufacture 'high quality synthetic' training data
There's this bizarre myth that higher-quality training data will magically fix the reliability problem. Why anyone thinks you can generate that data is a mystery. Maybe wishful thinking? Granted, there are times when we can usefully generate training data, but that usually involves adding noise. In image recognition, for example, you can multiply the number of labeled images you have by distorting them in various ways.
Nobody, at any level, has an actual clue what they are doing.
There's a sizable gap between what you see in the literature and what you see in press releases. That's nothing unusual. Not that industry and the tech press is entirely to blame, papers seemingly designed to generate misleading headlines aren't unusual either. It's the wild west in the field right now, but there is real work being done.
Re: (Score:2)
Indeed. LLMs are of limited use and a dead-end. They will become commonplace, but not very useful tools when running and creating them becomes cheap enough. But we will essentially not see more than they can do now. "Better search" and "better crap" is essentially it. Maybe, in some years, LLMs can even solve the SPAM problem, but that remains to be seen.
As to the next AI winter, yes, definitely. This must be number 4 or 5 or even higher. The field is doing it to themselves time and again by massively overp
Re: (Score:2)
Re: (Score:2)
I disagree on the base so trying to point out why:
>LLMs are not the same thing as AI
LLMs are AI. They may not be a GAI(general artifical intelligence), as there are many things they cannot do, but sayin LLMs are not AI is saying that a Ford model A was not a car because it could not do all the things we do today with cars.
>but when this stupid bubble bursts it will drag the whole industry down into another 20 year AI winter.
No, there will invariably be a bubble bursting, like we had in internet in 200
Re: Improvement in LLMs starting to plateau. (Score:2)
I just looked at the internets and the internet winter seems to have lastes 24 years and counting.
There may not be enough data in the world (Score:2)
and energy is running out fast too.
Surely this has to be the wrong way to go about doing artificial intelligence: AI companies throws exponentially-increasing amounts of resources at the problem and the amount of smarts they get out of their investment goes up more and more slowly. And privacy, IP laws and and the environment be damned to obtain those ever-dwindling advances.
A human brain burns about 20W. It's slow but it's infinitely better than any AI system on a Watt-per-smarts basis.
I think this was an
Too much energy indeed (Score:2)
[And each one costs around half a billion dollars in computing costs.]
I wouldn't just call this "crazy expensive". A Ferrari is crazy expensive. A yacht is crazy expensive. The American health care system is crazy expensive. This is ludicrous. Five hundred million dollars is the cost of keeping the lights on in New York City for two years.
For five hundred million dollars, I could own and operate 200 McDonalds franchises. I could buy the jet owned by Taylor Swift and fly it around the world a thousand
Re: (Score:3)
The American health care system, as corrupt and classist as it is, has a very large return on investment, it's not a cost but a benefit. Public healthcare is much more effective and far more ethical. Healthcare isn't a problem, but greed sure is.
Re: (Score:2)
The American health care system, as corrupt and classist as it is, has a very large return on investment,
The American "health care" system does indeed have a very large return on investment ... for shareholders. Not so much for patients. We spend twice as much per person as other nations which, unlike us, offer universal coverage and we still have worse outcomes! We're #54 in infant mortality, between Slovokia and Romania. We're #64 in maternal mortality, between Lebanon and Grenada. The West Bank (#61) is doing better than we are there.
Healthcare isn't a problem
Reality begs to differ.
Re: (Score:2)
of course, there are much better health care systems than the american one, thank goodness i'm not american
there are worse countries however
Re: (Score:2)
not to mention how many people are abusive trolls
Re: (Score:2)
I call bullshit on half a billion per training run. A training run takes a week. The costs are graphic cards, electricity and staff. They already own the graphic cards, so that cost is just depreciation. electricity doesnt' cost hundreds of millions per week, and the staff neither.
Re: There may not be enough data in the world (Score:2)
If 1 bowl of ice cream is good, 2 bowls must be better. Every billionaire knows that
Think of the all jerbs to build all those new nukular reactors we so desperately need. (Just in case
Re: (Score:2)
Well, it certainly is a feast of greed and stupidity. My biggest learning from the LLMs hype is how utterly without insight many people with real money are.
Re: (Score:2)
Human brains are likely the most efficient computing mechanism possible in this physical universe, within one order of magnitude or so. And then at least smart humans can do things we cannot explain (but verify to be correct), i.e. we do not even know whether mechanisms working within known Physics are sufficient to generate General Intelligence.
Who would win? (Score:2)
Comment removed (Score:3)
Re: (Score:2)
- Lack of Deliverables: Despite the funding, the project never produces tangible results or products, relying instead on promises and hype to maintain interest and secure more money.
OpenAI is expensive, but they have produced tangible results.
Re: (Score:3)
Just nothing that can actually meet investor expectations. A person might go so far as to call some of the claims we've seen outright fraud.
Re: (Score:2)
I would be willing to do so immediately. As an example, claims of "AGI" being imminent clearly qualify.
Re: (Score:2)
Re: (Score:2)
Yep. So common in human history that even an LLM can give you an answer that looks insightful. And the idiots that fall for it are always plenty.
Not this again. (Score:2)
GPT was a huge leap forward in a field that stagnated in the 80s, and seemed pretty dead in the 90s. Around 2000 we got some new breakthroughs with Bayesian belief networks, and then the train started rolling. And now people are disappointed that next year doesn't see AGI...
Seriously, that's more on the hype and the press releases, and the stickers that believe them, than on the field of AI. It's only natural that once you scale a mountain, you see a new range in the distance.
Yeah the problems are pretty bi
Re: (Score:2)
During the 2000s and 2010 there was surely a lot of advancements and "deliverables". I am still more impressed at the fact that AI translators can produce reasonably good output cross-translating across tens of languages than at their "production" of human-like speech.
Of course, I can understand the novelty that means for most people a computer that seems to exhibit rational thought and interesting dialogue. But, as many people here know, it's all smoke and mirrors... and I trust that people will eventually
Re: (Score:2)
I di agree that the NLP progressed of the last few decades are really impressive. But they have very little to do with AI in the sense most people think about AI. A lot gets hardcoded and a lot is from existing, human-made, translations. Also, while there are vast cultural differences, the essential experiences humans make in this world are still mostly the same.
Re: (Score:2)
GPT was a huge leap forward in a field that stagnated in the 80s, and seemed pretty dead in the 90s. Around 2000 we got some new breakthroughs with Bayesian belief networks, and then the train started rolling. And now people are disappointed that next year doesn't see AGI...
Fair enough, but you don't mention what is arguably one of the most important elements for training AI that became viable only recently: massively-parallel computing systems with very large RAMs.
And data. Don't forget the data. That stuff we get from the intarweb thingy.
Re: (Score:2)
Transformers were revolutionary. Bayesian belief networks, not so much.
But there are a few advancements in progress that will change the field over the next few years. Things like the integration of some of the more esoteric maths used in quantum mechanics that has applications in training AIs.
Oh, you're crackpot. Never mind.
Re: (Score:2)
But there are a few advancements in progress that will change the field over the next few years. Things like the integration of some of the more esoteric maths used in quantum mechanics that has applications in training AIs.
Oh, you're crackpot. Never mind.
Indeed. Somebody that thinks pouring some barrels of "Science" over the problem will magically manifest what he wishes for.
Re: (Score:2)
Things like quantum computing itself.
And there we can detect you are just spouting nonsense. Quantum Computing, should it ever work to a meaningful degree, is singularly unsuitable for AI computations. It is hard to imagine a mechanism that would be less suitable.
Re: (Score:2)
Did you miss the deep learning revolution of the 2010s?
I'd say that was the big breakthrough. ANNs basically went from a curio from history which under performed a variety of modern methods to absolutely blowing everything out of the water in terms of performance.
Machine learning was getting every more mathematical, slow, complex and borderline unimplementable. Within about 3 years we went from AlexNet being published to a good masters student being able to beat the absolute state of the art from 2011 in a
Fools, (Score:2)
just pay Elsevier, Springer and other gatekeepers of science several billions and access their huge stash of human knowledge.
You don't need more data than the total collective knowledge of humanity. Unless you're preparing to sell your products to aliens more advanced than us, rather than to mere humans.
Re: (Score:2)
That will go well. There is a reason LLMs get trained on simple things.
transmission of thoughts with language (Score:2)
Re: (Score:2)
That is a gross simplification and also inaccurate. Non-verbal thinking is a thing required in STEM. But the idea is very roughly correct. The thing is, if you talk meaningfully about something with another person, that other person need real insight into the topic for that to work.
LLMs cannot do "insight", the mathematics they use does not allow it.
Synthetic Data and Cow Brains (Score:3)
Bovine spongiform encephalopathy - Mad Cow Disease - is what you get when grind up dead cows and feed them to living cows. I suspect synthetic data will result in Mad AI Disease. Or an AI-centipede, as such mouth-to-ass configurations often produce. Let's give these AI's a few gigawatt-hours and a computer room. The synthetic data generators can pitch, and the AI's-in-training can catch. The offsping will probably want their own pronouns.
Re: (Score:2)
Mad Cow Disease doesn't help this discussion, it's nothing to do with what you're talking about. The phrase you're looking for does however exist and it's called "Model Collapse". It's not only been researched but we've even discussed it here before https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Ftech.slashdot.org%2Fstor... [slashdot.org]
The offsping will probably want their own pronouns.
That's an interesting way of signing off an otherwise thoughtful post. But signing off "- Insufferable arsehole" is shorter to write, just helping you make your posts more efficient in the future.
AI needs to leave school (Score:2)
We train humans in schools, but not just in schools. At some point, they need to go out and play, work, succeed, fail, and so on. That is how we learn to apply creativity to solve novel problems.
Perhaps the same needs to happen for AI in order for it to evolve. Up to now, it has mostly been stuck in the library, reading books. When it interacts with the world at large, develops an interest in solving problems that improve its own condition and that of others (human or AI) then I think we will see a new phas
Re: (Score:2)
We are talking LLMs here. LLMs cannot do that type of "training" because LLMs cannot determine whether they succeeded or failed at something.
Re: (Score:2)
We are talking LLMs here. LLMs cannot do that type of "training" because LLMs cannot determine whether they succeeded or failed at something.
Absolutely. But I still stand by my point. Maybe not LLMs, but AI generally, needs to get out of the (training) classroom somehow, and interact with the real world like humans do, and enjoy/suffer all of its ups/downs. What that looks like, I'm anxious to see. How you "motivate" AI to be curious about its environment and seek to improve it seems like an important step to achieve.
Power (Score:2)
Law of diminishing returns (Score:2)
I learned about the law of diminishing returns a few decades ago in Computer Science. I guess the OpenAI folks haven't heard of it.
Re: (Score:2)
They probably have. And they will probably know they are essentially dead. They have been trying to postpone the inevitable for more than a year now. Keeping the scam going is hugely profitable for some of the scummy individuals in there.
And even if it works... (Score:4, Insightful)
... it will not be that much better. The core components of insight and understanding will still be completely missing, as they require AGI and nobody knows whether that is even possible.
I think OpenAGI is basically simulating progress at this stage, because they cannot really make any in a meaningful way anymore. As they have produced nothing except huge losses so far, as soon as the pretty dumb investors realize what is going on, OpenAI is dead. Obviously, they want to postpone the time when that happens with grand claims about "AGI" and "reasoning" and other crap LLMs cannot do and will never be able to do because the mathematics does not allow it.
Shame they just can't (Score:2)
I know how to fix this (Score:2)