This is absolutely correct, and it's extremely important to understand that this behavior is baked into the model. It's not even so much that it's "baked into" the model as it IS the model.
Ever since the earliest days of LLMs and generative AI, we have always, always seen what people call "hallucinations." But the only thing that qualifies such output as a hallucination is that we, the human observer, has judged it as such. By construction, the model DOES NOT KNOW THE DIFFERENCE. If it knew or "understood," it wouldn't have produced the spurious output in the first place.
So every now and then, we see some amusing audio/video clip of some human managing to fool or "break" an AI agent, and it makes the rounds of social media for people to point at and laugh and say "tee hee, how funny!" And then these same people, without thinking about what these hallucinations mean and what they imply about the intrinsic nature of the technology, immediately turn around and ask Claude or ChatGPT what they should make for dinner, or who to vote for, or who to hire and fire; or worse, give them access to their personal data and critical systems. That is a level of uncritical thinking and cognitive dissonance that ought to be profoundly distressing.
From a functional perspective, there is quite literally zero distinction between an AI generated pretty woman breaking and suddenly speaking in Cantonese during an English-language interview, and an AI agent that violates constraints imposed upon it and causes damage. They are both "hallucinations" in the sense that they are unintended behaviors/outputs, and as we have already established, the only meaningful arbiter of "intention" is our own human judgment. AI proponents try very hard to make excuses and minimize the severity of the problem. But it is intrinsic to all models.
Whether through laziness or ignorance, when humans cede their authority to interrogate and decide the truth for themselves and let machines do their thinking for them, who or what ultimately becomes the authority for that truth, and who is or is not accountable for the consequences? Who has a vested interest in controlling the source of truth and knowledge, and what are their motives?