I've been a professional software engineer for 30 years.

AI has changed everything for software engineers.

I wrote a line of code by hand saturday (client blocks AI) and it was weird. I hadn't done it in months.

I don't have a crystal ball. I cannot say for sure whats coming.

But I would say this: if the scope of your ambition is small, e.g. the possible amount of value you can add as a business (or as an employee) is fundamentally constrained by something, then having a machine do that thing way more efficiently than you is a problem.

Machines will take an increasing share of your fixed pie.

Lots of software isn't that.

Lots of software is trying to make the pie bigger.

We're doing things now that we never would have tried. The incremental cost of writing code goes to zero. Software engineering changes. What assumptions about what you do and how you spend your time and what value you provide change when the cost of writing code goes down 10x or 100x?

For all of human history, the speed of acheiving every goal has been limited by available high-IQ-hours. At every moment in human history, there was unimaginable value not being created because it didn't make the cut compared to everything else that also needed doing - the brains capable of doing it didn't have the remaining hours to do it.

Artificial intelligence is changing the math on that. Unbounded level of available hours at what appears to unbounded level of IQ.

You are trapped in a prison made of your own lack of imagination.

(I'm not saying I'm not, by the way)

I don't understand your point. Can you explain it for me?

I assume you have never tried making an AI model of the type we're discussing (near-frontier LLM), and you have also never tried making a printing press.

Neither have I.

However, knowing a bit about how LLMs work, and knowing a bit about model training for non-LLM models, and having done a bit of word working and metal work, I assume it's actually easier now to make an AI model at home than it is to make a printing press at home.

My response included assumptions which might not hold.

If the way that police work actually worked was "we pick 100 names randomly out of the phone book, arrest them all, and 10 were innocent" ... the biggest problem would _still_ be that of a random sample you have a 90% guilty rate. That's actually not even civilization. Nobody actually trapped in a "society" like that would object to being detained and released.

So the way to think about it is, "we detain 100 people for whom we have probable cause - we believe a crime was taking place, and we believe they were proximal or involved in that crime.. and we have only a 10% false positive rate"

That's really quite good.

So for instance, if you have a riot and you arrest 100 people, and then later you find out that 90% of them are actually criminals, but 10% of them actually didn't break any law, and release that 10% - I'm fine with that. It was completely reasonable to arrest all 100 of them in the first place - they were, after all, in physical proximity to a riot, and they looked and acted enough like the 90% of actual rioters that the police (correctly) detained them.

A fine way of not getting arrested in this scenario is to stay the hell away from riots. And when "protests" become riots, _leave_.

If revolution, anarchy, and violence are the goal, fine. But then don't expect due process.

It sounds like you're saying that the printing press was just a technical enabler, sometimes in the hands of bad people. You're saying that the printing press per-se was neither good nor evil; it was the people who used the printing press for evil ends that really caused the problem.

Me too.

>Should we accept jailing 10 innocent people to catch 90 actual guilty criminals?

Yes.

This is one of those questions that seems philosophically interesting but is not interesting pragmatically.

When you jail someone, you have reason to believe they are proximate to what you believe is a crime. You remove their freedom to move/act temporarily while you ascertain the existence/severity of a crime, and if they are a party to that crime.

When you figure out that you made a mistake, you release them.

The (presumed) alternative here is patently ridiculous.

A society where 0 innocent people are arrested but 90 guilty people run around is much worse than a society where 90 guilty people are caught and 10 innocent people are temporarily detained.

The actual #1 threat to Americans is not being improperly arrested, but being victimized by known repeat criminals who are repeatedly released back into society.

Bukele shows what is possible.

think "Manhattan Project"

The USG doesn't want to be significantly behind anyone else when it comes to having ASI. That's the national security _interest_.

Our elected officials didn't know anything about atomic bombs in 1942 either. They still don't. Back then, the USG provided national security controls and oversight to one of the biggest and most successful fronteir research and engineering projects in human history.

>I'm certainly not sold on the idea

I'm not either.

The Aschenbrenner paper got into my brain and nothing else has dislodged it yet.

tl;dr:
1) if China developers acquires ASI significantly before the US does, the national security posture of the American people and homeland becomes untenable.
2) the disastrous lax security and processes within the American frontier labs make it inevitable that any AGI/ASI developments that happen there will rapidly be exfiltrated to China
3) The last time there was a national security research project this serious was the Manhattan project.

The point of nationalization isn't to keep huggingface from getting hacked or to keep self driving cars from running over strollers. Nobody who matters cares about any of that stuff. We are all expendable to them.

The point of a US national security takeover is to try and delay or prevent ASI breakthroughs being leaked to adversary countries.

>There are no misuses of the printing press, this is just your grotesque failure of logic.

Every governing authority that has come after the printing press disagrees with you. All have concluded that the printing press is misused often enough that they have jailed or killed people who they have deemed to have misused it.

I knew what I was doing when I put "misuse" in quotes.

>But you've already given us reason to completely disrepect your opinions.
Who is this "us" you speak of?

>Right, it's the printing press responsible for wars, not christianity.
The printing press was the amoral technology that enabled the reformation and the "democritization" of Christianity, and was a direct contributor to the reformation wars.

A different testcase points to this: The great schism between Rome and Constantinope (Roman Catholic vs. Orthodox) happened 500 years earlier than the protestant reformation, but did NOT lead to hundreds of years of fratricide between east and west.

>"National security reasons" is a justification for controlling applications of AI, not AI itself.

You actually didn't read (or understand) the paper that you dismissed.

Aschenbrenner argues that the reason to nationalize AI labs is because the AI labs are badly run, from an opsec perspective. It has nothing to do with good or bad applications of AI. He argues that it's trivially easy for someone to exfiltrate AGI/ASI from the labs as they are currently run; the research culture being used is not compatible with the national security impact of the work being performed.

We are at the cusp of something that is likely to be more dangerous and more impactful than the atomic bomb. Look at what USG did once the Manhattan project got under way. New cities were built in remote locations. The top researchers under American jurisdiction were forcibly relocated.

I don't know what the USG/NS response will or should look like. It probably doesn't look like "Sam gets to keep doing whatever he wants"

>The printing press was invented at a time when safety standards were virtually non-existent

I have no idea if the machines were dangerous to the operators or not, but whatever deaths may have happened to the operators is a rounding error compared to the deaths that are directly downstream of how the widespread usage of the printing press changed civilization forever.

The latter is actually my point: not that printing presses are unsafe, but that the way printing presses changed society caused a lot of death.

For the printing press, I think generally people have believed that the benefits outweigh the drawbacks.

This will also be the case with AI.

The headline is reasonable.

AI is a technology. It doesn't have moral agency. It can be used for good and for bad.

So far, the printing press is responsible for more deaths than AI.

(Printing Press -> Mass distribution of Luther's bible translation -> Protestant Reformation -> hundreds of years of religious wars in Europe)

Despite all of the death that is downstream of the printing press, the usual response in the west has been to ban specific "misuses" of the printing press - not the printing press itself.

The specifics that Altman is saying - that voluntary slowdowns are good, or that external oversight over OpenAI is bad - I disagree with.

Instead, I agree with Aschenbrenner's paper. The AI labs should probably be nationalized for national security reasons.

(Notably, Aschenbrenner claims he was fired by OpenAI for expressing these concerns internally)

If the cops receive an anonymous tip that CowboyNeal is an insensitive clod, and then decide to arrest him without checking out any of the evidence, I blame the cops. There's plenty to bitch about with Flock but "the cops don't care about doing their job" isn't one of them. That's a cop problem.

Don't treat any stupid robots as somehow "better" than an anonymous tip called in from a payphone. It's just, like, something's opinion, man.

We now live in a fully connected world where most users are connected through web browsers and are the intended victims.

This particular story is an exception to that. Remote software, accessed through a web browser, normally does not have read/write access to your local filesystem. But people are granting local filesystem access to proprietary AI services.

I think that granting such access is .. wow. I wouldn't do it, though I can see how if you use a specialized account for it, and if you're confident that your OS has no local user priv escalation bugs, it might be ok. In your sandbox you trust.

But GP's comment is about whose decision that is. If you use Linux, then it's your decision. If you use iOS, then it's Apple's decision. And MacOS is becoming less Linuxy and more iOSy every year. I think it makes plenty of sense for stories like this to result in people dumping MacOS for Linux. (But daaamn, their values surprise me. You wanna make your computer less secure instead of more? Oh well, like Rumsfeld said, freedom is untidy!)

No, if the state were renamed, it would be to Old America. NM might not be the true original, but if you look at it through the eyes of European colonization and continuous governance, then New Mexico has the best claim of the fifty. (Though if you take a less racist view, then I guess Alaska had the real OGs.)

Randi McGinn, a lawyer for New Mexico, said applying the full $5,000 penalty to the number of violations would create too large of a penalty under the US Constitution's protections on due process.

I ass/u/me this refers to the Eighth Amendment, but I don't remember seeing any monetary constants encoded within that text.

Google's telling me Meta is worth about $1.8 trillion so how would a mere $40B be "excessive?" It's only about 2% of their worth! Civil penalties in excess of 2% of someone's worth are routinely accepted as constitutional, aren't they? If this case were about me driving drunk and killing a whole family, I bet no one would flinch at the 8A implications of taking 2% (!?!) of my assets, and everyone would laugh at my lawyer if he brought up 8A for such an insignificant fine.

I understand why we have 8A, and everyone deserves mercy, but I don't see why Meta deserves exceptional mercy as a singular special case that we would be horrified to see applied anywhere else. Bringing up 8A for such a tiny fine would be giving Meta a very unique carve-out that we normally would not apply.

It may be that $5k is too much to charge criminals per act, in that each crime didn't really cause that much loss, but bringing up due process concerns seems like bullshit when the fine is already so small. NM and Meta had been trying to negotiate the cost of illegal business for years before negotiations failed and it went to trial. Meta decided to let a jury decide, and the jury's come up with a fairly tiny fraction of their assets.

If Meta thinks the price of illegal business is still too high, then they can lower the number of willful violations they commit per year, to optimize the advantage they gain from illegal actions at an affordable rate.

Geez, Zuck, just do less of it. If $40B seems like a lot, then maybe try only $10B worth of crime next time. What's the big deal? Live and learn. After this little fine you still have 98% of the company left to go on and incur the next wave of liabilities.

This is pretty standard when someone leaves Microsoft.

The management above the person leaving sends a nice message about them and their work and NEVER tells the truth about _why_ they are leaving.

At the upper levels I don't think there's really any sincerity about anything said, one way or another. This is just how corporate America operates.