> but "not understanding cost" is a dangerous assumption to make.

One of the best, most succinct phrases in this discussion. Thank you.

We don't have to wait for AGIs to develop their own agency and build terminator robots. Industrial robots with no agency and with auditable and deterministic programming already kill humans. The industrial robot doesn't understand there's a human in the safety cage with it. It doesn't understand that moving through space occupied by a human has a cost. It doesn't know there's a human there and it doesn't know what a human is.

Designing automation that can never kill a person is already an unsolved problem with standard, non-intelligent technology.

I don't know if AI makes it harder or easier. On one hand, AI behaves, in effect, non-deterministically (even though at some level it is deterministic, it's not observably so).

OTOH, in organic intelligences, adding sensors, adding alignment, giving feedback - tend to produce humans who accidentally kill less frequently. So maybe something like the 3 laws will conceptually be part of the System Prompt of future AIs, and they'll figure out how to avoid HAL 9000 problems.

Thanks for the interesting response.

The strongest man today; the hardest workers today... are not 10x or even 2x as strong or as physically productive as the best in human history. Depending on what you think about ancient man, we are possibly weaker and dumber than some of our ancestors.

The transmission of new ideas across memory, time, and place is the extra term in the equation of human achievement, civilization, and prosperity. It's not that labor doesn't matter. It's that ideas are like compound interest. Labor adds principal to our civilizational account balance. But Ideas compound it.

In the long term, ideas are all that matter.

I have the same reservations you do - powerful people will continue to want to corner and concentrate their power.

It's unclear how well they will be able to do that, however. The capability lag time between a paid frontier model and a self-hostable open-weights model is months.

The ability to train a good-enough model will also be commoditized. In my youth, we had distributed crypto challenges.... to get super-computing work done on mass democratized hardware of like-minded individuals. So in the future, we can fall back on analagous distributed model training efforts if some better alternative doesn't exist.

I am skeptical about a post-scarcity society and I am skeptical about man's "labor" ever going to zero. Of the infamous twelve possible AI outcomes, I am somewhere in middle. For biblical reasons, I believe the extinction of man is not going to happen, and I believe the utopia of man is also not going to happen.

Control of other people's speech is always about asserting power.

It's never actually about "reducing harm"; it's about asserting power.

Talking about it in "harm" terms is a clever hack: ... because most humans are sensitive to harm/harm reduction along their moral reasoning (see: Jonathan Haidt)

But "Never hurt my feelings" is an impossible bar, and it doesn't scale across a civilization. It cannot be a basis for law.

The attempt to abuse the "harm receptors" of political enemies is contributing to the mutual dehumanization across faction lines.

After all, women go for more decidely female-related professions such as female doctor, female surgeon, female attorney, or female director.

Of course the AI versions would get paid less.

This. No good game was ever created from a single prompt. Not to an AI, not a human development team. Every single person writing those damn "look which game my AI built from a single prompt and $x in tokens!" postings is a newbie and almost certainly has never actually shipped a single game.

Game development can be summed up as endless iterations. Every system you build, every visual you create, every sound, text, button, movement, weapon, skill, powerup needs polishing, refinement and changes. It is never, never good enough the first time.

No it is not, but it IS a very successful platform for a specific subclass of games. If I were in it for the money, I could probably AI refactor some of my earliest games and publish them on Roblox and make a quick buck. If your thing is the kind of games you played on your C64 back in 1793, then Roblox got you covered. Well, if you ignore the scams and money-grabbing.

*sigh* I wish Unity would actually complete some of their dozens half-finished sub-systems instead of constantly chasing the latest trend. It's become really, really annoying. The engine used to be really good. These days, it falls apart if you look at it the wrong way, and at least from the responses to my bug reports it's clear that fixing issues is way down on the agenda.

Maybe this is their attempt to regain grounds in the indie game market - the very market they pushed away by focussing on AAA requirements for years and ignoring the single devs and small teams. If so, I can already tell them that it won't work. We collectively laugh about all the "look what my AI built for me from a single prompt" postings on reddit, where people proudly show off some AI slop that would get them last place at a game jam.

Their AI efforts so far: Tried to build and sell their own AI tools, to a collective yawn from the audience. Built a barely-working MCP server, then deprecated it before it was even finished in order to push a CLI interface to the editor instead, which only works if it's run as the same user as the editor on the same machine. And completely ignores every single advise on workflows and good development practices you can think of.

Unity, I love your engine, I despise your management. Please return to when you were actually good and stop chasing butterflies.

The target audience for this particular rule of the AI Act isn't the producers or users of AI. It is the consumers of content that might be AI generated.

And frankly, letting people know if they see AI content or not is a fair thing to ask.

Fun fact: The EU AI Act considers this. It makes a difference between "AI generated" and "AI modified".

But it's hard to put this into the type of watermarking they are using. Because if you generate a text and then change a few things here and there what you are doing is degrading the watermark. But since the watermark is probabilistic, it's hard to tell if it was a weak watermark from the start or if it was degraded.

We use Hubspot. Here's what we use it for:

- the "live chat with support person" bubble on our web application
- single funnel for incoming/outgoing support tickets & support communications with our customers

Adopting it was an old decision.

We would love to not use hubspot; the service delivered vs. the ongoing costs seem like a bad deal now.

At some point, we'll get around to migrating to something else.

I don't think that makes the document any less your own than having a human copy editor conduct a similar review which has been done since forever.

Which is what I've been saying as well. If all you're doing is asking for suggestions to something you have already written, using AI is no different than using a human editor. It's your work. It's been written. You're just looking for advice on how to improve it.

This is completely different than people who tell AI what they want and it does all the work.

That ain't the same thing.

This thing is doing things you didn't tell it to. It might even do things you explicitly told it to not do. It is not a case of "well, you could've forseen this as a consequence if you had been careful" like your campfire. My example is only a bit silly, it is absolutely possible that an AI agent will decide to hack NASA because you asked it for some data for a school project. Even in the broadest possible sense, that does not fall into stuff that the law so far has considered to be in the domain of negliegence.

AI agents do stuff you could not possibly have expected. Heck, I asked Claude a fairly simple knowledge question yesterday and it wanted to read from /dev/null. There is no argument you can make that would convince a jury that I should in any way have expected that and taken precautions. In a hypothetical world where a read from /dev/null randomly kills a puppy somewhere on the planet, I don't think I would've been convicted.

That nightmare scenario actually did play out not long ago near me. A woman from out of state was falsely ID'd by an AI camera of some kind and was detained for months and it wrecked her life.

My sentiment is that the process was wrong and that the officers and other parties involved in this should all be fired, and the woman compensated.

None of that is what normally happens; that is not happening "millions of times". That's why it was newsworthy when it _did_ happen.

The original comment was philosophically open ended.

I made the example in my response very specific because arguing about hypothetical generalities is usually not productive.

The difference is this: it's difficult to ascertain if someone is guilty or innocent without arresting them. YOu don't arrest people via a random statistical sampling; you arrest via a biased selector: You arrest people you have high odds of beleiving are likely to be guilty. And you arrest them because you cannot ascertain their guilt via non-destructive observation. Without arresting them, they are allowed to simply walk away while you are trying to ascertain their status.

An arrest rate of 10% of all humans would be ridiculous.

A false positive rate of 10% _among the population of people you arrest_ is acceptable.

So the applicable comparison isn't "your cell phone doesn't work 10% of the time", the applicable comparison is "when your cell phone wasn't working, the tech support team guessed correctly as to the root cause 90% of the time, but guessed incorrectly 10% of the time"

IOW, 90% success rate with a 10% FP rate amongst a problematic cohort.

>No I maintain it would be pathetic performance by LEO if they were willy nilly arresting people hoping to bag the perp.

Nobody is advocating for that.

The cops should only be arresting people they have a plausible reason to arrest. If it turns out that 10% of the time their guess was incorrect, that's acceptable. They are not performing random arrests and getting a lot hit rate. They are doing targeted arrests and getting an ultra high hit rate.

>AI is being forced inappropriately into peoples lives with no opt out *everywhere*

I agree with you, and I also don't like that, and I think stopping that via market choices and legislation, as appropriate, is the right thing to do.

In my mind that's totally separate from what I am talking about and commenting on.

That said, you can make legislation so that you're not forced to use it (or be spied on by it), but you cannot make a law that stops AI development. If nothing else, the law won't reach beyond our borders. And eventually a nation hostile to you will develop ASI and then they will impose it on you (and worse)

I've been a professional software engineer for 30 years.

AI has changed everything for software engineers.

I wrote a line of code by hand saturday (client blocks AI) and it was weird. I hadn't done it in months.

I don't have a crystal ball. I cannot say for sure whats coming.

But I would say this: if the scope of your ambition is small, e.g. the possible amount of value you can add as a business (or as an employee) is fundamentally constrained by something, then having a machine do that thing way more efficiently than you is a problem.

Machines will take an increasing share of your fixed pie.

Lots of software isn't that.

Lots of software is trying to make the pie bigger.

We're doing things now that we never would have tried. The incremental cost of writing code goes to zero. Software engineering changes. What assumptions about what you do and how you spend your time and what value you provide change when the cost of writing code goes down 10x or 100x?

For all of human history, the speed of acheiving every goal has been limited by available high-IQ-hours. At every moment in human history, there was unimaginable value not being created because it didn't make the cut compared to everything else that also needed doing - the brains capable of doing it didn't have the remaining hours to do it.

Artificial intelligence is changing the math on that. Unbounded level of available hours at what appears to unbounded level of IQ.

You are trapped in a prison made of your own lack of imagination.

(I'm not saying I'm not, by the way)