I think we should make a distinction between "AI" and "AGI" here. Human intelligence consists of a number of disparate faculties -- spatial reasoning, sensory perception, social perception, analogical reasoning, metacognition etc. -- which are orchestrated by consciousness and executive function.
Natural intelligence is like a massive toolbox of cognitive capabilities useful for survival that evolution has assembled over the six hundred million years since neurons evolved. The upshot is you can reason your way to the conclusion that you can remove the blue block from underneath the red block without disturbing the red block, but then metacognition uses *other* mental faculties will overrule this faulty conclusion. Right there you have one big gap between something like an LLM and natural intelligence (or AGI). LLMs have just one string to their bow, so they can't tell whether they're "hallucinating" because detecting that requires a different paradigm. The narrow cognitive capabilities of generative AI means it requires and engaged human operator to use safely.
For many decades now I've heard AI advocates claim that the best way to study natural intelligence is to try to reproduce it. I always thought that was baloney: the best way to study natural intelligence is to examine and experiment with animals that possess it. What AI researchers try to do is to write programs which perform tasks that previously could only be done by humans. That's why when any AI technique starts to work, it's not AI anymore. But these tasks are so restricted, and the approach taken is so uniformed by actual psychological research, that I doubt that these limited successes tell us anything about natural intelligence.
Until, maybe, now. Now that AI has reached a point of unprecedented impressiveness, I think it teaches us that reductive approach to AI we've been taking won't generate systems that can be trusted without human oversight. That doesn't mean these systems aren't "AI" by the field's own Turing test.