I use a definition I've extrapolated from the Turing Test:
In maths, if f(x)=g(x) for all x, then f=g
However, for humans, f(x) isn't a razor-thin line, it's a band that follows a normal distribution, so we have to modify this a little. AIs also don't produce an absolute result but a band that, again, follows a normal distribution. (Nobody demands identical neuron firings between humans for the same stimulus, or even by the same human for the same stimulus.)
If the band for f(x) predominantly lies within 1 std dev of the mean value of g(x) at that point, then, functionally, f(x)=g(x)
Ok, that's closer. But we haven't defined x, f, or g.
Let's extend this out. Let x over an interval be the steps during any non-random conversation. f and g are how the AI and the human mind progress during that conversation. Let x, overall, cover ALL the conversations, regardless of who with, held over some non-trivial period of time (say a year). This period has to be long, because what you want to measure is not how much the person remembers but how much the person's brain (and therefore responses) have altered as a result of each conversation, because that is a key aspect of intelligence. Memory is not nearly as important as the change in the model itself, and you need sufficient time to model those changes.
Why measure a continuity? Because intelligence is about rationally proceeding through a chain of thought, not instantaneously having a thought. An AI can certainly be capable of that, and LLMs do indeed proceed, but do they proceed in a manner that we can map within the known bounds of intelligence? That is the important question. Individual thoughts and decisions aren't important, sequence is.
There is a second aspect, which is orthogonal to this, which was raised by Dr John H Conway and, independently, by Professor Roger Penrose, which is of free will.
Conway proved, mathematically, that you cannot achieve free will through anything that is deterministic OR random, that free will has to be a fundamental property of a particle or of physics itself if humans are to possess it. He did not show they did possess it, he only showed that if free will exists, it has to be found in something within physics itself.
Penrose's argument is actually not that dissimilar in that he, too, argues it has to be a fundamental property of physics. He argues that, for free will to work, you need retrocausality - the collapsed state of any quantum system MUST precede the collapse itself in some manner. That's not actually too difficult in QM, as time is not fundamental but an emergent property and therefore the chronological order is merely what we observe and has no relationship whatsoever to the events themselves.
So far, Penrose's argument is unproven (unlike Conway's), but we have to consider the possibility that he is correct. If he is correct, then LLMs won't do the job, you'll need a quantum computer that therefore possesses the precise property needed.