This is just inaccurate. I did not say you committed an error within logic. I pointed out that we have no scientific means of identifying consciousness, and that you can't use logic effectively if you can't identify unprovable states. Everything else you've added is outside the scope of what I said, and kind of makes my point for me.

The problem with your "proof" is that it is logically accurate. "If an AI can be abused by cruel language, then using it is unethical because it's operating in a condition of slavery." Logically sound. Combine that with the opposite and make the excluded middle - congrats - you've just demonstrated why logic doesn't work for this entire domain of problem.

In order for your argument to work, you have rely on the cultural assumption that because slavery is unacceptable between humans, it is necessarily bad in all domains; but "slavery" is entirely subjective. As an example: you cannot logically prove whether or not keeping a pet dog is "slavery" or "unethical"; the answer depends on how the dog experiences it. Technically, we have no way of actually proving whether AIs (or even dogs) have experiences, and if so, whether or not "slavery" is a problem for them. Those are subjective problems that require details logic can't provide.

Strictly speaking, the only thing I'm claiming here is that the TOS is logically indefensible, but so are the objections to it.

Well ... that's kind of true. Plenty of people like simple answers that cut off curiosity, which is fine, and that's one way to look at what I'm doing.

It might have been more flattering to say that I'm really curious about why some rocks and water eventually evolved into human brains, and where and how consciousness showed up. The utilitarian "is AI conscious right now" is enough for some folks - I'd be a better suited to a chat with folks who know the hard problem isn't solved but wonder if watching this develop with an open mind might turn out to be instructive.

You might want to brush up on reading comprehension, seeing as every assumption you've made is not one that I've asserted.

Who said anything about both ways? If you can't understand the concept of "unprovable" you can't do logic at all.

Is it? We have no means to test for this.

While I'm inclined to share your opinion about rocks, it begs questions like: what about plants, jellyfish, worms, roaches, fish, cats, dogs, horses, whales, monkeys, that guy down the street you don't like very much ..? (That last one about the guy down the street sounds like hyperbole in the current context, but how often have you heard someone gripe that a third party is "whining about nothing"?)

If you've solved the hard problem of consciousness, publish it. Otherwise, the rational observation is that we don't actually know how it works, and in the meantime, encouraging people not to act like psychopaths just because they can might be a good idea in general.

Several generations of people spent most of their lives getting acclimated to the idea that businesses want whats best for their customers and contracts are not just legal agreements but genuinely efforts to collaborate agreeably.

A lot of folks are still in a continuous state of shock about the fact strangers might lie to them.

If only there were some way people could maintain healthy sleep cycles based on physical reality instead of being biologically bound to the made up numbers on a clock.

For practical purposes, no - they can't. The blind-spot is akin to poppy plants and heroin. We're mostly wired to think that as long as a person is making a choice, the problem is primarily about holding people accountable for their decisions. That keeps us almost entirely blind to systems that hijack our decision making processes at either the individual or social structure level. It means we blame the addict for their dumb choice, and maybe a little the economic system that encourages addiction, but no one feels the immediate need to grab a spear and hunt down the poppy plant the way we would if it were as obvious as a tiger running around eating people.

As a threat vector, the optimal position for AI as a whole system is the same as the position of the poppy plant. Specifically not smart enough that anyone would ever consider it a threat, completely dependent upon a "human in the loop" so that we always blame the person and never the AI, and yet perfectly adapted to hijack our brains and economic incentives, even without needing to know that those things exist.

That's insufficient to solve the problem. It's not incompetence, or malfeasance, or reliance on AI. It's incompetence, malfeasance and reliance on AI.

We need to do a much better job of understanding and dealing with the fact that those factors are socially drawn together, metaphorically, like magnets within the context of default human behavior. We won't get anywhere debating which magnet to blame. We need to actually take the magnetism itself into account, and solve for it.

Ooh. The heck with debate - what have you learned?

I partly agree. I do worry that most of people aren't recognizing a critical detail about the scope of evolutionary processes.

If a person dies of a heroin overdose, two things are true:

1. 1. They made an unfortunate decision to take heroin, that resulted in their own death.
2. 2. They lost an evolutionary battle with a syringe.

The second statement is, socially, a weird way to describe it. It is also accurate with regard to the scope that evolution actually covers in the universe.

We could probably get by just fine not thinking about syringes as an evolutionary competitor that occasionally kill some of us: we got through most of the industrial era without noticing that evolution does not distinguish between self-replicating organisms and other phenomenon like bicycles or languages. But I doubt we'll be able to continue much longer if we can't even recognize that our evolutionary competitors are not just animals or germs, and the playing field is not just the resources themselves.

Human minds, decision making processes, social structures, all of that are part of the environment, and literally every pattern in the universe that repeats is in on the evolutionary struggle. AI isn't just agents on computers. It's also the entire phenomenon, including the part where a human brain decides whether to devote energy toward building what the AI "wants" or what we actually need.

If we die out because we get so caught up building data centers that we don't handle other potential risks to our survival, most people will probably say "that wasn't AI, that was us making stupid decisions." But the only relevant fact at the evolutionary scale will be that, just like a heroin addict chasing a syringe, we will have failed to notice that our addiction is a survival strategy for a competitor perfectly adapted to avoid us even noticing that it is a threat.

"No true scotsman?" Okay.

The actual underlying observation, unadressed, is that evolutionary processes don't care about human interpretations about what counts as alive or intelligent or any of that. Richard Dawkins popularized that observation when he coined the term "meme," and while I'm aware that you'll be tempted to call that an appeal to authority, doing so doesn't address the phenomenon itself. As a phenomonon in the universe, AI is evolving and competing for resources. Human minds and social structures are two aspects of its ecosystem. Saying "but that's not AI, that's us making bad decisions" does not change the outcome, but it does leave you blind to the mechanisms by which evolving phenomenon can pose a threat.

I keep seeing arguments that the "AI Will Kill Us All" argument is a bunch of doomsaying by people who overestimate the potential intelligence of these systems. My own intuition leads me to argue that AI does not have to become self aware or autonomous. It just has to be sufficiently adaptable to make us both dependent and stupid. Today my confirmation bias feels vindicated.

Ugh. C'mon.

All it has to do is be sufficiently adaptable and profitable to convince us that we'd rather keep building data centers than care about whether everyone has enough water and energy. It does not have to become a self-aware superweapon. It just needs to be a mind-virus that causes us to make stupid decisions at scale.

It looks like they are glossing over the actors' concern that producers will use them as stars in movies where they did not actually perform. Even if AI can't slop a convincing movie from scratch today, given the way greed works, we could take it as a given that producers will basically be looking for ways to use famous people's faces without having to pay them by any means they can get away with from now on out.