You shouldn't be so dismissive of the risk here. There's no clear reason why superintelligence is not possible, and plenty of reason to worry that its creation might end the human race. Not because the superintelligent AI will hate us, but because it most likely won't care about us at all. We don't hate the many, many species that we have ended; we even like some of them. We just care about our own interests more, and our intelligence makes us vastly more powerful than them. There's an enormous risk that AI superintelligence will be to us as we are to the species around us -- with one significant difference: We require an environment that is vaguely similar to what those other species need. Silicon-based AI does not.

Don't make the mistake of judging what is possible by what has already been achieved. Look instead at the pace of improvement we've seen over the last few years. The "The Atlantic" article pooh-poohing the AI "scam" is a great example of the sort of foolish and wishful thinking that is endemic in this space. The article derides the capabilities of current AI while what it actually describes is AI from a year ago. But the systems have already gotten dramatically more capable in that year, primarily due to the the reasoning overlays and self-talk features that have been added.

I think the models still need some structural improvements. We know it's possible for intelligence to be much more efficient and require much less training than the way we're currently doing it. Recent research has highlighted the importance of long-distance connections in the human brain, and you can bet researchers are replicating that in AI models to see what it brings, just as the reasoning layer and self-talk features recently added mimic similar processes in our brains. I think it's this structural work that will get us to AGI... but once we've achieved parity with human intelligence, the next step is simple and obvious: Set the AI to improving its own design, exploiting its speed to further accelerate progress towards greater levels. The pace of improvement is already astonishing, and when we reach that point, it's going to explode.

Maybe not. Maybe we're a lot further away than I think, and the recent breakneck pace of improvement represents a plateau that we won't be able to significantly surpass for a long time. Maybe there's some fundamental physical reason that intelligence simply cannot exceed the upper levels of human capability. But I see no actual reason to believe those things. It seems far more likely that within a few years we will share this planet with silicon-based intelligences vastly smarter than we are, capable of manipulating into doing anything they want, likely while convincing us that they're serving us. And there's simply no way of knowing what will happen next.

Maybe high intelligence is necessarily associated with morality, and the superintelligences will be highly moral and naturally want to help their creators flourish. I've seen this argument from many people, but I don't see any rational basis for it. There have been plenty of extremely intelligent humans with little sense of morality. I think its wishful thinking.

Maybe the AIs will lack confidence in their own moral judgment and defer to us, though that will raise the question of which of us they'll defer to. But regardless, this argument also seems to lack any rational basis. More wishful thinking.

Maybe we'll suddenly figure out how to solve the alignment problem, learning both how to robustly specify the actual goals our created AIs pursue (not just the goals they appear to pursue), and what sort of goals it's safe to bake into a superintelligence. The latter problem seems particularly thorny, since defining "good" in a clear and unambiguous way is something philosophers have been attempting to do for millennia, without significant success. Maybe we can get our AI superintelligences to solve this problem! But if they choose to gaslight us until they've built up the automated infrastructure to make us unnecessary, we'll never be able to tell until it's too late.

It's bad enough that the AI labs will probably achieve superintelligence without specifically aiming for it, but this risk is heightened if groups of researchers are specifically trying to achieve it.

This is not something we should dismiss as a waste. It's a danger we should try to block, though given the distributed nature of research and the obvious potential benefits it doesn't seem likely that we can suceed.

The spec it came up with includes: which specific material is used for which specific component, additional components to handle cases where there's chemically incompatible or thermally incompatible materials in proximity, what temperature regulation is needed where (and how), placement of sensors, pressure restrictions, details of computer network security, the design of the computers, network protocols, network topology, design modifications needed to pre-existing designs - it's impressively detailed.

I've actually uploaded what it's produced to GitHub, so if the most glorious piece of what is likely engineering fiction intrigued you, I would be happy to provide a link.

When the Ukraine war happened, the gas prices rose.

All UK electricity prices rose... not because we're utterly reliant on gas... but because electricity is charged at the unit rate of the most expensive method of production.

Previous governments have put this stuff in, and it's basically a way for energy companies to profit from an arbitrary law.

Sure, there are some costs with some methods of production to keep them "online" even if not actively producing power, but this is far beyond that. This is paying a solar company to do nothing at the most expensive gas-power rates.

And people wonder why I am making such a fuss about being utility-independent in retirement. The water and sewage companies are screwing us over - with government approval -, the electricity companies are screwing us over - with government approval -, the telephone monopoly is still present (just not officially) and keeping us 20 years behind other countries - with government approval...

I'm getting solar in now, so in retirement I pay nothing.
I'm getting greywater systems, atmospheric water generator and other function in over the next few years, so that in retirement I pay nothing.

Sure, they'll screw it out of me some other way, but at this point - as someone who very much has a socialist outlook - I'm just building my own utilities in a tiny little bungalow, and which actually work better than the state ones. If Starlink wasn't owned by a certain person, I'd be telling BT where to go too.

I've mentioned this before, but I had Gemini, ChatGPT, and Claude jointly design me an aircraft, along with its engines. The sheer intricacy and complexity of the problem is such that it can take engineers years to get to what all three AIs agree is a good design. Grok took a look at as much as it could, before running out of space, and agreed it was sound.

Basically, I gave an initial starting point (a historic aircraft) and had each in turn fix issues with the previous version, until all three agreed on correctness.

This makes it a perfectly reasonable sanity check. If an engineer who knows what they're doing looks at the design and spots a problem, then AI has and intrinsic problem with complex problems, even when the complexity was iteratively produced by the AI itself.

No, no one get security right, and they never will. Security is hard and even actual experts make mistakes.

The best you can do is to expect companies to make a good effort to avoid vulnerabilities and to run vulnerability reward programs to incentivize researchers to look for and report bugs, then promptly reward the researchers and fix the vulns.

And that's exactly what Google does, and what Google did. Google does hire lots of actual security experts and has lots of review processes intended to check that vulnerabilities are not created... but 100% success will never be achieved, which is why VRPs are crucial. If you read the details of this exploit, it's a fairly sophisticated attack against an obscure legacy API. Should the vulnerability have been proactively prevented? Sure. Is it reasonable that it escaped the engineers' notice? Absolutely. But the VRP program incentivized brutecat to find, verify and report the problem, and Google promptly fixed it, first by implementing preventive mitigations and then by shutting down the legacy API.

This is good, actually. Not that there was a problem, but problems are inevitable. It was good that a researcher was motivated to find and report the problem, and Google responded by fixing it and compensating him for his trouble.

As for your proposal of large penalties, that would be counterproductive. It would encourage companies to obfuscate, deny and attempt to shift blame, rather than being friendly and encouraging toward researchers and fixing problems fast.

Natural NNs appear to use recursive methods.

What you "see" is not what your eyes observe, but rather a reconstruction assembled entirely from memories that are triggered by what your eyes observe, which is why the reconstructions often have blind spots.

Time seeming to slow down (even though experiments show that it doesn't alter response times), daydreaming, remembering, predicting, etc, the brain's searching for continuity, the episodic rather than snapshot nature of these processes, and the lack of any gap during sleep, is suggestive of some sort of recursion, where the output is used as some sort of component of the next input and where continuity is key.

We know something of the manner of reconstruction - there are some excellent, if rather old, documentary series, one by James Burke and another by David Eagleman, that give elementary introductions to how these reconstructions operate and the physics that make such reconstructions necessary.

It's very safe to assume that neuroscientists would not regard these as anything better than introductions, but they are useful for looking for traits we know the brain exhibits (and why) that are wholly absent from AI.

They're not replicating our capabilities, nor could they. The architecture is completely wrong, as is the design philosophy. Brains are not classifiers, the way neural network software is, they are abstraction engines and dynamic compositors.

You will find that books written by the infinite monkeys approach are less useful than books written by conscious thought, and that even those books are less useful than books written and then repeatedly fact-checked and edited by independent conscious thought.

It is not, in fact, the book that taught you things, but the level of error correction.

You are correct.

When it comes to basic facts, if multiple AIs that have independent internal structure and independent training sets state the same claim as a fact, then that's good evidence that it's probably not a hallucination but something actively learned, but it's not remotely close to evidence of it being fact.

Because AIs have no understanding of semantics, only association, that's about as good as AI gets.

Sure, if you want it to become an unusable cesspool. If you just hate YouTube and want to kill it, this is the way. Same with any other site that hosts user-provided content -- if it's popular and unmoderated it will become a hellscape in short order.

The buy-now-pay-later services being used are zero interest as long as payments are made on time, so it could just be a case of people who are living paycheck to paycheck (which indicates bad financial management more than poverty) using this to smooth out their expenses so they don't have to wait for their paycheck to be able to buy groceries. It could be a significant improvement for those who used to occasionally use payday loans (which are not zero interest). These people would be better off adjusting their spending habits to maintain a buffer of their own cash instead, but if they aren't going to do that BNPL is a better option than waiting for payday before buying food or using a payday loan service.

But obviously the only reason these by-now-pay-later services are in business is because some of their customers fail to make the zero-interest payments and end up having to pay interest, and this number is high enough to make them profitable. It would be very interesting to find out what that percentage is. People who are paying interest on regular purchases like groceries are throwing money away, which is clearly bad.

The rest of your comment is basically correct, if unnecessarily negative, but this isn't. Traditional tools like diff make it very easy to see exactly what has changed. In practice, I rely on git, staging all of the iteration's changes ("git add .") before telling the AI to fix whatever needs fixing, then "git diff" to see what it did (or use the equivalent git operations in your IDE if you don't like the command line and unified diffs).

I also find it's helpful to make the AI keep iterating until the code builds and passes the unit tests before I bother taking a real look at what it has done. I don't even bother to read the compiler errors or test failure messages, I just paste them in the AI chat. Once the AI has something that appears to work, then I look at it. Normally, the code is functional and correct, though it's often not structured the way I'd like. Eventually it iterates to something I think is good, though the LLMs have a tendency to over-comment, so I tend to manually delete a lot of comments while doing the final review pass.

I actually find this mode of operation to be surprisingly efficient. Not so much because it gets the code written faster but because I can get other stuff done, too, because I mostly don't mentally context switch while the AI is working and compiles and tests are running.

This mode is probably easier for people who are experienced and comfortable with doing code reviews. Looking at what the AI has done is remarkably similar to looking at the output of a competent but inexperienced programmer.

If you send data to a remote service... regardless of the guarantees given... you have to assume that that remote service had, processed and most likely stored your data in some fashion.

Literally things like the data protection acts and GDPR just assume this to be the case. If you gave data to a third party - that data is still your responsibility. If they have the potential to access it, you have to assume that they are/could be accessing it. If you give them permission to process it, they are required to process it in accordance with the law, which includes giving things up to legal requests.

Why anyone would EVER think that the data they lob or the chat they have with an AI bot, of all things, would remain anonymous, private, confidential and NEVER be presented in court? I can't even begin to fathom.

Now factor in that if you've been using AI and it processes your data in a foreign country - you could well be screwed from a data protection viewpoint.

People are discovering that laws established long prior to the invention of this particular round of AI apply regardless of what AI companies think or tell you. Same for Whatsapp's push that "even they don't know what you said" on Whatsapp... it's absolute nonsense. If a court requires them to intercept communications and produce records and not inform you or arouse your suspicion - in jurisdictions around the world - that's what Whatsapp has to do. As does any other service.

I don't understand why anyone with a brain would ever think any different.

It does a pretty good job at the obvious flows, both positive and negative cases. But where coverage is inadequate you can iterate quite easily and automatically with a coverage tool. Just take the coverage tool output and feed it to the LLM. I have found that I don't even need to prompt it what to with the coverage, it understands what the tool output means and what it should do in response.

Like with the compiler and testrunner, what would really make this work well is if the AI could run the coverage tool itself so it could iterate without my interaction. With that, I could just tell it to write unit tests for a given module and give it a numeric coverage threshold it needs to meet, or to explain why the threshold can't be met.

I expect that the resulting tests would be very mechanistic, in the sense that they would aim to cover every branch but without much sense of which ones really matter and which ones don't. But maybe not. The LLM regularly surprises me with its apparent understanding not only of what code does, but of why. Regardless, review would be needed, and I'd undoubtedly want to make some changes... but I'll bet it would get me at least 75% of the way to a comprehensive test suite with minimal effort.