The hype around AI in education at least makes sense. If you're impressed with a chatbot and try to use it to learn about something you don't know much about, it can seem like an impossibly patient and wise tutor with limitless knowledge. The illusion is incredibly convincing... until you try having it teach you about a subject you know very well.
The terminally delusional, sadly, still assume that the current state-of-the-art must be good enough for primary school children as the material isn't terribly complicated. The reality, as you know, is that it doesn't matter. AI doesn't operate on facts and concepts, but on statistical relationships between tokens. It will still generate nonsense and even defend the nonsense it generated. It's hard to image a resource that's worse for a struggling student.
Just for fun, this is what 4o produced when asked a simple question about fractions:
> is this correct? ab/c = a/c b
No, the equation \( \frac{ab}{c} = \frac{a}{c} b \) is not correct as written. However, if you interpret it as follows:
\[
\frac{ab}{c} = a \cdot \frac{b}{c}
\]
that would be correct. You can factor \( a \) out of the fraction on the left side, which gives you:
\[
\frac{ab}{c} = a \cdot \left( \frac{b}{c} \right)
\]
So, the correct representation should maintain the order and context of multiplication and division.
When I asked simple follow-up questions, it generated even more nonsense. I can't even being to image the kind of confusion a response like this would cause a struggling student.