DeepMind Co-Founder Proposes a New Kind of Turing Test For Chatbots 84
Mustafa Suleyman, co-founder of DeepMind, suggests chatbots like ChatGPT and Google Bard should be put through a "modern Turing test" where their ability to turn $100,000 into $1 million is evaluated to measure human-like intelligence. He discusses the idea in his new book called "The Coming Wave: Technology, Power, and the Twenty-first Century's Greatest Dilemma." Insider reports: In the book, Suleyman dismissed the traditional Turing test because it's "unclear whether this is a meaningful milestone or not," Bloomberg reported Tuesday. "It doesn't tell us anything about what the system can do or understand, anything about whether it has established complex inner monologues or can engage in planning over abstract time horizons, which is key to human intelligence," he added.
The Turing test was introduced by Alan Turing in the 1950s to examine whether a machine has human-level intelligence. During the test, human evaluators determine whether they're speaking to a human or a machine. If the machine can pass for a human, then it passes the test. Instead of comparing AI's intelligence to humans, Suleyman proposes tasking a bot with short-term goals and tasks that it can complete with little human input in a process known as "artificial capable intelligence," or ACI.
To achieve ACI, Suleyman says AI bots should pass a new Turing test in which it receives a $100,000 seed investment and has to turn it into $1 million. As part of the test, the bot must research an e-commerce business idea, develop a plan for the product, find a manufacturer, and then sell the item. He expects AI to achieve this milestone in the next two years. "We don't just care about what a machine can say; we also care about what it can do," he wrote, per Bloomberg.
The Turing test was introduced by Alan Turing in the 1950s to examine whether a machine has human-level intelligence. During the test, human evaluators determine whether they're speaking to a human or a machine. If the machine can pass for a human, then it passes the test. Instead of comparing AI's intelligence to humans, Suleyman proposes tasking a bot with short-term goals and tasks that it can complete with little human input in a process known as "artificial capable intelligence," or ACI.
To achieve ACI, Suleyman says AI bots should pass a new Turing test in which it receives a $100,000 seed investment and has to turn it into $1 million. As part of the test, the bot must research an e-commerce business idea, develop a plan for the product, find a manufacturer, and then sell the item. He expects AI to achieve this milestone in the next two years. "We don't just care about what a machine can say; we also care about what it can do," he wrote, per Bloomberg.
Obvious bullshit (Score:5, Informative)
Almost all humans cannot turn $100k into $1M and those that can either have pre-existing wealth or criminal networks or luck. Which makes this "test" much worse than the original Turing test. I suspect the guy knows that, but bullshit like this will probably draw greedy but not very smart investors in.
Re:Obvious bullshit (Score:5, Insightful)
In other words a rich arsehole completely devoid of humanity. He'll do well.
Narcissist (Score:2)
That's the correct word. The Venn diagram of salespeople, CEOs and narcissists is essentially a big circle.
Re: (Score:2)
Indeed. Unfortunately. Makes DeepMind deeply suspect though.
Re:Obvious bullshit (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
This makes him sound like one of those people who only see the economic value of other people. In other words a rich arsehole completely devoid of humanity. He'll do well.
Sounds like a candidate for office to me.
Re: (Score:2)
Yep, one nice psycho to determine the fate of a nation.
Re: (Score:2)
Re: (Score:2)
Almost all humans cannot turn $100k into $1M and those that can either have pre-existing wealth or criminal networks or luck. Which makes this "test" much worse than the original Turing test. I suspect the guy knows that, but bullshit like this will probably draw greedy but not very smart investors in.
Nah, he just wants someone to give him the bot to test so he can start getting 10x returns on his money.
Re: (Score:2)
If an AI that could turn $100k into $1M existed, the experiment would not be repeatable. Either it was a fluke, or whatever it did will be copied and quickly stop working.
Re: Obvious bullshit (Score:2)
I feel like it'll quickly pivot to offering courses in e-commerce drop shipping.
Re: (Score:2)
Indeed. But morons with money do not get that.
Re: (Score:1)
Re: (Score:2)
Re: Obvious bullshit (Score:2)
Unless there's a time limit, it's pretty straightforward. Put it in an index fund and wait about 25 years.
Re: (Score:2)
And hope your luck doesn't run out in the last year, as it did in 2008. Or 1929.
Re: Obvious bullshit (Score:2)
That just delays it by 5 years.
Re: (Score:2)
If the new pseudo-"Turing Test" discussed in TFA is applied, I doubt they would make any extended time allowance. They're already prepared to brand anyone who's not financially capable as a non-human.
Re: Obvious bullshit (Score:2)
Most tech startups turn $100M into $0 in less than five years. I can get an AI to do that pretty easily too.
Re: (Score:1)
Re: (Score:2)
It is bullshit in multiple ways
RE his statement: "It doesn't tell us anything about what the system can do or understand, anything about whether it has established complex inner monologues or can engage in planning over abstract time horizons, which is key to human intelligence,"
First of all, not all humans even have an inner monologue. There has never been any comprehensive studies done to even determine how many people do or do not. And anecdotally, many otherwise very successful people claim to not have
Re: (Score:2)
I agree on all of that, except that the current AI is not AGI and cannot be. If we ever make a technological implementation of AGI, it is well possible we will notice too late though. But nobody knows whether that is even possible.
Re: (Score:2)
ChatGPT doesn't even approach the capabilities of a crow let alone a human being. At best it can regurgitate something someone else said or make modifications to that which come across
Re: (Score:2)
And why are you so dismissive of humanity? Honest people can turn that initial investment into a million dollars in a number of ways, from careful investment to simple compound interest.
Sure, it's a bad test, but not because your view of humanity is even remotely accurate.
Re: (Score:2)
My view of humanity is spot-on and it is the established fact in the respective psychology and sociology areas. It is not popular because it tells most people that they are nowhere as smart and insightful as they think they are. This becomes even worse when the one telling them that is somebody that makes the cut. Denial, ridicule and other stupid but typically human reactions are sure to follow. Interestingly it seems to only weakly correlate with IQ, i.e. there are a lot of high-IQ morons.
Re: Obvious bullshit (Score:2)
Re: (Score:2)
My dad's friends' favorite joke: "How do you get one million dollars? Give $mydad'sname two million dollars!"
Comment removed (Score:4, Interesting)
Re: (Score:2)
Meaningless in this context. Otherwise a bank-account would have AGI.
Re: (Score:3)
Re: (Score:2)
Bank manager Hjelmman died in 1904. He left a testament and 15 000 Finnish marks (about half of the budget of a city during that time). Instructions in the testament said that people need to wait 300 years before money can be used. Until then it should sit in a bank collecting interest. Now, after about 100 years, the value of that money is about 4% of the original. After few decades there won't be anything left.
This happens, because bank interest rate is lower than inflation rate, so money loses value over
Re: Obvious bullshit (Score:3, Insightful)
I turned my salary into 1M in 30 years. There are 20 millions Americans like me. Significant chunk of population. We are not rich arseholes, we are victims of inflation
First millionaire lived in US in the beginning of XIX century. His million would have been 25 million dollars. The ones with 1 million dollars now are not millionaires, they are schmucks who can't even retire on this Zimbabvian dollar amount
Re: (Score:2)
Does not qualify. Do you have trouble understanding the conditions given in the story?
Re: Obvious bullshit (Score:4, Interesting)
If this is the people working on AI we are doomed. Statistically, even doing bad business will get you to 10x. I propose a different test: if a panel of real normal humans can not tell if a person has empathy, they will be considered a machine, regardless of biology and mechanics. Actually, a similar test is proposed in the movie Bladerunner.
Re: (Score:2)
harsh as it sounds, we eventually will need something like that to reduce the concentration of wealth.
Re: (Score:2)
ChatGPT already passes this test. Hasn't it already created enough hype to turn $100K into $1M? Basically this "suleman test" is a hype bot test. Any bot that can create enough hype will pull in "angel money".
Trivial to game (Score:1)
Just use the original Turing Test. (Score:4, Insightful)
Using the fake modern Turing Test is like evaluating someone's ability to run a marathon by testing their speed in a 100 m sprint. There still isn't any chatbot that can pass as human for long, at least if we consider trolls a separate species.
fucking lol (Score:2)
One of the most profitable human enterprises in history is slavery. You want the bot to pass the turing test by pretending to be human? Just let it act unethically.
What Turing Had In Mind (Score:5, Interesting)
When Turing proposed the Turing Test he imagined that a capable AI system would require only about 250 megabytes of memory. Why would he suppose such a ridiculously small amount of memory would be sufficient? I do not know how he would have answered, but one fact that might have weighed in his thinking is how little human speech a human needs to hear to reach a high level of competence. Up through the age of 6 a human has only heard about 200 megabytes of text equivalent.
He surely never imagined "AI" system that sweeps up a terabyte of human speech, and thus have 30,000 years of human speech to copy by statistically directed remixing using a random number generator to create the appearance of intelligent speech. The ability to do this nullifies the foundation of the original Turing Test. An LLM can easily find in its vast collection of narratives something "likely" to repeat with some alteration to create a false appearance of intelligent speech to a human interrogator.
LLMs make the Turning test obsolete, and so some other measure of intelligence has to be devised. When you can copy (to a rough approximation) anything anyone has ever said, simply producing intelligible speech means nothing.
Re: (Score:3)
When Turing proposed the Turing Test he imagined that a capable AI system would require only about 250 megabytes of memory. Why would he suppose such a ridiculously small amount of memory would be sufficient? I do not know how he would have answered, but one fact that might have weighed in his thinking is how little human speech a human needs to hear to reach a high level of competence. Up through the age of 6 a human has only heard about 200 megabytes of text equivalent.
If so, Turing seriously misunderstood how humans learn speech.
For one thing, rather than just passively ingesting a stream of text human children are deliberately taught with a specific objective of understanding vs memorization. This is a lot more data efficient than typical ML systems.
But far more importantly, this speech is coupled with many, many terabytes of secondary data. Audio tones, physical feedback, and of course copious visual input that helps them understand the meaning of the text.
And as predi
Re: (Score:3)
Re: (Score:2)
Also, children don't consume text before they can read; they listen to speech, which has much more information in it than text. You can tell if someone's angry by listening to them talk, even if you don't know the language being spoken.
Re: (Score:2)
These language models aren't intelligent, they're just well spoke parrots. That's wh
Re: (Score:2)
I do not see that LLMs make the turing test obsolete.
I haven't seen anything about chatGPT that indicates to me it could fool me for any length of time into believing it is a real human being with real life experiences and insightful opinions/advice/understanding about people and situations going on in my life, for example. Or the ability to understand and consider scientific ideas outside regurgitating "the mainstream" narrative.
Re: (Score:2)
ChatGPT is not preprompted to pretend to be a human. It would have to be specifically prepared for that particular task, there are too many protections right now. As for "ability to understand and consider scientific ideas outside regurgitating 'the mainstream' narrative" - what percentage of humans do you think qualifies for that? AI doesn't have to be smart to pass Turing test, it just have to sound slightly smarter than average citizen. I think we passed that threshold already.
So many arguments anti-AI
Re: (Score:2)
> ability to understand and consider scientific ideas outside regurgitating 'the mainstream' narrative" - what percentage of humans do you think qualifies for that?
Just about anyone. I can tell someone an idea or theory and then ask them to explain it back to me, and then their thoughts about it, what questions they have about it, why they think it is is not mainstream consensus, how it compares with what they were taught, and so forth.
From what I've seen chatGPT will just spit back a bunch of consensus
Re: (Score:2)
Existing LLM's such as the one that drives ChatGPT prove that it is evidently possible to generate coherent sentences in a language by randomly producing output based on the statistical frequency of tokens appearing near eachother in the training texts.
Intelligence, therefore, cannot be reasonably deduced based on yjr output of a system alone, but the process by which that output is achieved must also be examined. Intelligent creatures do not generally communicate by randomly spitting out verbage in th
BFW (Score:2)
Re: (Score:2)
Re: BFW (Score:2)
That's not how taxes work.
It'd be 400k profit (1,000k - 100k investment - 500k to platform).
You'd keep 70% of that or 280k for 180k net.
I personally work a significant amount of time for 180k before taxes and would definitely find use for it.
If I had one of those AIs (Score:2)
I would then have all the money.
At least, 1000 trillion dollars of it.
Assuming I could still convince the AI to give the money earned to me, since I pressed its start button, and also asked nicely.
Understanding? (Score:2)
This guy is in charge of an AI empire but I question his grasp of the turing test.
It isn't just mimic human to fool human. It is supposed to be self aware, conscious of its own simulated gender and such.
Like a guy pretending to be a woman; aware of issues a woman might have biologically, socially with some other abusing dolt, raising a child and such. Most men would fail at this task unless relying on stereotypes strongly and talking to another guy that is inept and relying on the same stereotypes.
This is
Re: (Score:2)
I question his grasp of the turing test.
It isn't just mimic human to fool human. It is supposed to be self aware, conscious of its own simulated gender and such.
Um... you really need to read Turing's paper [oup.com] again.
Re: (Score:2)
One of us might need to read the paper again. I don't think it is me.
If you get beyond the first paragraph that you linked, it goes into conversation about hair length and sexual identiy. How many AIs are concerned/aware of their hair length? It implies some understanding of style, not simple image matching, but 'style' and how some hair styles and lengths might fit better on a man or woman.
Again...not patern matching or 'turn 100,000 into 1,000,000' ... but attempting to think like a human.
Re: (Score:2)
Your claim, in case you've forgotten:
It isn't just mimic human to fool human. It is supposed to be self aware, conscious of its own simulated gender and such.
Like I said. You really need to read the paper again. Maybe get someone to help you.
Re:Understanding? (Score:5, Informative)
> I question his grasp of the turing test.
Yes.
"According to Turing, the question whether machines can think is itself “too meaningless” to deserve discussion (442). However, if we consider the more precise—and somehow related—question whether a digital computer can do well in a certain kind of game that Turing describes (“The Imitation Game”), then—at least in Turing’s eyes—we do have a question that admits of precise discussion.
"Suppose that we have a person, a machine, and an interrogator. The interrogator is in a room separated from the other person and the machine. The object of the game is for the interrogator to determine which of the other two is the person, and which is the machine. The interrogator knows the other person and the machine by the labels ‘X’ and ‘Y’—but, at least at the beginning of the game, does not know which of the other person and the machine is ‘X’—and at the end of the game says either ‘X is the person and Y is the machine’ or ‘X is the machine and Y is the person’. The interrogator is allowed to put questions to the person and the machine of the following kind: “Will X please tell me whether X plays chess?” Whichever of the machine and the other person is X must answer questions that are addressed to X. The object of the machine is to try to cause the interrogator to mistakenly conclude that the machine is the other person; the object of the other person is to try to help the interrogator to correctly identify the machine. " - Stanford Encyclopedia
"I believe that in about fifty years’ time it will be possible to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning" - Alan Turing
The Gambling Test? (Score:2)
Is there a reverse Turing test? (Score:3)
A bit off-topic, but has anyone a devised an AI that can tell if it's conversing with a human or not?
I was just thinking you could pit it against a chatbot via a GAN, and they'd both steadily improve ...
Re: Is there a reverse Turing test? (Score:2)
It's not AI, but a captcha is a reverse Turing test.
Most humans will fail, while chatbots will ace (Score:2)
In terms of valuation, most humans will fail this test. Most investors today are putting their money on these "AI chatbots", hoping to turn their $100K into $100B. No investor is really putting their money on a human being. If a chatBot can speak for itself, it is the real hype creator and therefore the real billionaire.
A troll based CAPCHA (Score:1)
I propose a test where trolls identify genuine humans by devouring them and identifying genuine humans by flavor and crunchiness.
Re: (Score:1)
Screw that. Test for happiness (Score:2)
A true test of intelligence is an AI that maximises pleasure and/or contentment.
Idiots make money every day. Idiots run massive companies. Trump ran a country and he is still a piece of shit with massive problems.
Truly intelligent people don't need to prove themselves or seek validation.
If anyone invents an intelligent AI the last thing it would want to do is participate in this test.
Well that's a silly test. (Score:2)
The old Turing test is still fine (Score:4, Interesting)
The old test still works. A reasonably sophisticated human can distinguish human responses from AI-generated responses 100% of the time, if they are given a few minutes to ask multiple questions.
If there is any flaw in Turing's original test, it's this: Turing did not consider the fact that some human beings are very naive and suggestible. If the tester is naive enough, even a very simple chatbot (like Elisa) might seem "human", I suppose.
I've played with a few chatbots (including ChatGPT), and they are always defeated by a simple cognitive test (it's based on an item in the St. Louis University Mental Status exam [SLUMS], which is a test used to screen for dementia). First, you tell the chatbot a simple story which is about a paragraph long. Then you ask it simple questions to test whether the chatbot "understood" the story. The chatbot always gets most of the questions wrong.
Re: (Score:2)
Can you provide the specific example of such text and questions? (if I'll find my own and get different results, you will claim it was too easy) I don't believe in _most_ of the questions wrong. Some yes, but over 50%, unlikely, for 1 paragraph text, for latest version of ChatGPT.
Re: (Score:3)
If you google "SLUMS", you can find the test online. (The SLUMS is a rare example of an "open source" cognitive test, so it's free to use). It's question #11.
I didn't use the exact "story" from the test, I just made up a similar one. And, yes, chatGPT did get *some* of the answers correct (maybe it was even 50%), but it missed very obvious questions which no human being would miss (unless perhaps they had severe dementia or poor English comprehension).
I don't know much about AI. But I get the impression
Hahaha (Score:2)
Profitability is not intelligence! (Score:2)
Intelligence is many things [wikipedia.org], but the ability to generate profit is just one small application of a few of them; it's an optimization for our current economic system. How do Einstein, Tesla, MLK, the Dalai Lama, and most teachers fit on this metric? Shouldn't we aspire to AI capable of their accomplishments? It's almost as if Suleyman is asking for a real-world test of Asimov's laws of robotics [wikipedia.org].
Is this a finite parameter game? (Score:1)
if smart (Score:2)
Haha. Good one. Puts into practice the adage "If you're so smart, why aren't you rich?"
Turn $1M to $100k would be a better test (Score:1)
To test this, I asked one AI "How would you waste 900.000 dollars". The core of the answer was
"I can suggest using the money to invest in a good cause, such as supporting charitable organizations that address important issues like poverty, education, health care, or environmental protection. Alternatively, one could use the money to travel the world, learn a new skill or hobby, or start a business ventures. "
Those are ways of spending money, not wasting it, dear AI!
Re: (Score:2)
"Wasting" requires a value judgement AI is incapable of making. To a pattern-matcher, 'wasting' and 'spending' are equivalent. ...And you could find a lot of humans who would look at that list and agree it's 'wasting'.
Re: (Score:2)
Charity is mostly waste of money. If you have a million people in hunger and you give them food,, soon you have 2 million people in hunger. They even did a test with rats where they removed all the predators and gave them never ending source of food. At first the rats multiplied but eventually they stopped caring about their kids and the population died out.
Also most businesses will fail, so starting a business is also usually waste of money.
I would also argue that most of the skills and hobbies people want
Paperclip Maximizer (Score:2)
Seems like Nick Bostrom's paperclip maximizer would be even better as a modern Turing test. Give the AI $100K as seed money and incentivize it to make as many paper clips as possible. If we all die, the AI passed the test.
What? (Score:1)