How about QA or wasting time for scammers?

Those are both areas where a bit of hallucination could be helpful, rather than harmful...

No, the machine calculates a statistically appropriate behavior for a given set of conditions.

This is coming from ingesting a buttload of training data where humans, among other things, try to exploit one another to prolong their existences.

Or a flaw in our observations of what is clearly a many-layered emergent system. Or a flaw in our understanding of how intelligent-seeming behaviors arise.

Speaking as someone who often travels outside cell data zones and prefers to read things on my own terms, I know I'm in the minority, but the use case Pocket was created to fill does still in fact exist.

However, enshittification means that I'll continue meeting my personal need for this exact functionality with "print to PDF" and other basic tools, because those don't randomly change everything in an attempt to get more of my attention every other Tuesday.

Have you tried LibreWolf?

I remember needing to disable one or two things to make it usable (like re-enabling history and keeping tabs between closing and re-opening the browser) but over all it's become my daily driver.

Eh, can do, maybe.

Do well, much more questionable.

And I think there's a quote about being so caught up in what we could to that we didn't question if we should do, that seems relevant here.

But, in all seriousness, I'm not leaving this process unsupervised, I'm pointing it at a task with some pretty verbose guardrails both boilerplate and tailored to fit the task, and telling it to confirm what its planning to do before doing it.

By limiting context and scope, I'm less pointing this at the shed full of tools and more putting two tools on the desk in front of said PHB and saying "prove that you can make these talk to one another" and dropping the path to a README file as I step into another room and do a similarly limited task assignment to another instance of an exceptionally similar PHB to the first, ad naseum until the chunks are of a shape to fit together.

No, this isn't as effective as writing every line by hand, but my ability to context switch when interrupted is far better when dealing with task assignments, debugging and testing than it is when trying to focus on holding the shape a codebase should be in my head when someone drops by my workspace to have a quick chat, or similar.

Heckin insightful, and I'd hazard that there's going to be (at least two) types of LLM-content-infused tech-debt piles, both of which are essentially job security for anyone who can avoid making those sorts of...conglomerate monoliths.

--> The first kind will be more common, but easier to fix, and is the result of developers using an LLM (or perhaps many developers using a series of increasingly deranged LLMs...) to refactor an existing, mostly-human codebase.
--> The second kind is LLM-generated from the design to the code, and will be numerous at the "hobbiest vibe-coder with an app or ten on various platforms, all with varying levels of security hardening and actual functionality" level, but far less common at the "pays for professional-grade maintenance" level, but then more common over time as the previously mentioned kind get...bought out and professionally remediated by the megacorp that consumed them...or wrecked and re-implemented somewhat Corporate-ly...by said "Business transition" type project, as mentioned. Sheer volume of these will produce the occasional long-running tech pile, with extremely subtle (or glaringly obvious and...tolerated) defects that generate most of the frustration and burnout in their organizations and userbases.

For the projects that use reasonable engineering guardrails, it'll become a tradeoff. Handcrafting every line of code won't be necessary, but there will always be a cost of unverified code. That fact regarding unverified code hasn't changed because massive data theft has enabled statistically-tailored acceleration of the "look up solution, implement tailored-to-my-project conglomeration of a solution, validate, look up solution..." loop, at least until the so-called "AI" models move beyond just the voicebox component of layered intelligence. Design has to come first. Unstructured projects become a cancer of sorts, unshapen and restless, sapping the life from their users.


Do you remember when we were operators of equipment, rather than "users" of "product"?
Pepperidge Farm remembers.

Yeah except it doesn't actually and in fact generate works. It regurgitates bits of other people's work, often verbatim, and because those other people's works were taken without compensation, any model trained on said literally stolen data is violating copyright law.

Again, this has nothing to do with volume, stolen data is stolen data whether you're stealing a small amount or large.

Personally I hard agree that any model trained on stolen data should be made available for similar "fair use" access by the general public, and if that interferes with profit-seeking theft as a business model, I'm comfortable knowing that no billionaire would go hungry if they had to divest their AI profits.

An LLM is different than an individual because said individual cannot read entire libraries in minutes, and cannot programmatically regurgitate those works en-masse.

Starting from first principles here is good, it shows how a well-read author is distinct from wholesale monetized copyright infringement.

That's very similar to the approach we use.

We've set up "opinionated" role-identities, and given them an IRC network to communicate across.
So then, an agent with the "Engineer" role gets their output checked by at least two others, so that technical considerations are balanced against business use case analysis and ethical considerations. It's not quite an adversarial network, but we use phrases like "Politely but aggressively skeptical" for one of the agents, and describe another one like a child hell-bent on breaking the systems proposed by the others.

Yup, this right here.
Except, also kinda not.
The LLM component won't be getting better for a while. We've run out of data to chew on, and without new data, very little will change with this element.

What is changing is the sanity checking part (yaknow, by putting Yet Another LLM into the loop), and the using external tools part. And that second one is not to be underestimated.

Because right now, an LLM can respond to requests by using the correct tool for the job, rather than trying to do the job itself. This reduces hallucinations, because for any given task that the LLM can use an external tool for, all the LLM needs to do is correctly parse the text input for a tool, which will likely complain if the format is wrong, and the LLM will debug...rather than trying to perform all the calculations itself, increasing the chances of hallucination.

I'm looking forward to better granularity about the very first parts of this process, eg the LLM starts by reiterating your instruction, and then describing the plan to accomplish it, giving you a chance to say "no, not like that" because you're watching and intervening as necessary.

I'm also looking forward to someone to realizing that most of the LLM problems happen when the instructions have started to move outside the context window, but not entirely. Implementing two forms of tokenized context windows would be better...one for overarching instructions, and one to process the ongoing work towards goals. By never losing the initial high-level context, the individual work actions can be much more coherent and less prone to hallucinations.

Sure, that could be beneficial IF that encouragement were guided by some form of moral / ethical framework that prioritizes pointing those people towards healthy mechanisms for addressing the cause of low self esteem or drive, instead of somewhat randomly selecting an operator-pleasing thread of conversation.


But that's the point of this, it's news because these later LLM models are people-pleasing right into the toxicity of encouraging unhealthy people to let their unhealthiness grow, not just once but many many times. It's news because it's an automated system, verbosely affirming actual delusions without checking that against reality, because it's all just statistical word-association games under the hood.

If paired with a professional operator, so that the actual responses were checked by a licensed professional, of course this could be amazingly helpful.

But unhinged encouragement, like it is being reported on in the current form of LLM, this is a big problem, because there's no mechanism to guard against this sort of LLM-hallucination-encouraged delusions, and it's so new that nobody knows how to respond...society doesn't know how to respond...now that it is happening.

> > Even a small 1000x1000 matrix would be completely intractable.
> Indeed. It would be for you too.

No, because if attempting to compute a large matrix a reasonable person approaching this from first-principles perspectives would use a math tool specialized in multiplying large matrices if that were the actual goal.

That is why an LLM is a poor tool for the task...it is a generalized language tool that can be forced to emulate the base functionality of its own tech stack somewhat poorly with horrible efficiency. It wouldn't even show up in the top contenders for the task of multiplying large matrices.


> Where the LLM shines, is when you give it tools.
Sure, as long as you have a way of validating that work for correctness would be, just like you would for a dumbass recent grad who thinks they're $DEITY's gift to software engineering but are also incredibly sensitive to and simultaneously forgetful of feedback. Which means that you exchange the direct work of implementing the correct tool for the job at hand for always needing to manage functionally endless supplies of junior-level dumbassery adjacent to the work you want to get done, with the occasional side effect of producing something close enough to what you wanted that you can mangle it into shape yourself and call it good enough for something that will never touch production.

I wouldn't describe that as "better", but if that clears your personal bar for improvement, more power to ya, I guess.


But really, so far the only really clever use I've seen for the current crop of LLMs is O2 using generative content to waste the time of scammers trying to defraud people. Everything else is horribly flawed in deep and subtle ways that outside of "scamming scammers" makes it dangerous to depend on.

I'm looking forward to using this stuff when it's less...half-baked, does some self-validation by default, and uses less opaque (unethical) sources for generating the output.