Earlier this year, important clues to the physical basis for consciousness have been found (and none of these are in current AIs). Recent experiments have shown that consciousness has a quantum basis.
1)Experiment #1: Xenon quantum anesthetic effects
It has been found that Xenon gas acts as a very effective anesthetic. It has been found to work by disturbing quantum effects in microtubules. Interestingly, isotopes of Xenon (which are chemically identical but have different quantum effects) were not as effective as an anethetic. This indicates that anesthesia (Xenon at least) disrupts consciousness via quantum effects, not chemical effects as one would have normally expected. Start with this article, but there are many others now, especially this year from reputable journals.
https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Fwww.nature.com%2Farticle...
2)Experiment #2
Microtubules in the brain have been shown to have quantum effects. This was very unexpected as quantum entanglement has only been shown in very small molecules, whereas microtubules are very large. Not only that, but the larger the microtubule, the larger the quantum effects, which is not what had been found with other molocules. Roger Penrose (Physicist) and Stuart Hameroff (Neurologist) have been involved in a lot of this work (#1 above and #2).
https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Fwww.sciencedaily.com%2Fr...
https://ancillary-proxy.atarimworker.io?url=https%3A%2F%2Flegacy.cs.indiana.edu%2F...
In short, the proposal (now backed with experimental evidence) is that consciousness arises from quantum effects in neurons, in particular microtubules. Disrupting these quantum effects (with certain Xenon isotopes) also disrupts consciousness. At that point, the subject is as inanimate as a chair or a shirt, just an object with no senscience.
Given the physical mechanism of consciousness is now starting to be known, it should be possible to manufacture machines that do exhibit similar quantum effects in their processing units, which will make them 'conscious' just as we are, like in the film A.I.
At that point, we'll have to think about these things, but with the current A.I.s using classical computation on classical CPUs..no they are not conscious at all.
Why did 'consciousness' evolve this way? I am not sure, but I am guessing that the computational power of quantum computing in organisms is far more efficient than classical computing (think brain weight, energy requirements), so it evolved, and 'consciousness' is a side effect of it.