My experience, mainly with generating SQL queries, is that AI inevitably gets it wrong multiple times, so what I have had to do is more of a kind of meta-programming; giving the model cues and corrections. I have created some pretty sophisticated SQL queries, but there's no way in hell I can just pop the first go-around into my code and have it run. Either it's outright faulty code that will fail, or it's just not producing the correct results.
Now SQL is a fairly limited and ring fenced language (excluding stored procedures of course). I've never tried it with a general use language, but I imagine those problems will get more pronounced. That's not to say it might not be useful for translating natural language specs into code, but if my experience with SQL is any indicator, it's going to require a lot of massaging. There's probably still productivity boosts to be found here, which will likely have in effect on the number of programmers out there, but to me, it feels more like a layer of abstraction that will require a different kind of programming, rather than replace programming.
As an example that isn't coding, I have been building models for what I expect is a government procurement next year. This involves taking previous Requests For Qualifications documents, updating them with current knowledge of government expectations, procurement rules, and so forth. Again, building these model RFQs is an iterative process, not simply one of "Take these RFQs from previous procurements, update them with this new information I've uploaded, and give me model RFQs based on these premises I will provide." My test run took about three or four hours of a kind of conversation, where I correct and shape, understanding the cues the LLM needs to produce the desired result, and the better I get at understanding not just the kind of information and cues the LLM requires, but the most effective means of "encoding" that information, the more efficient the LLM is at producing the desired results.
That sure sounds like programming to me, albeit at a much higher level of abstraction. LLM, at least where it stands, is just another platform, a very powerful one, but as with all programming languages, the larger the command set and the more complex the lexical structures, the more room for bugs, and the more subtle some of those bugs can be.