It's not quite that simple, and that's why this is so strange.
These HCI commands are in the FreeRTOS binary blob.
That's not true. I've actually developed products that use ESP32s. I've even done some reverse engineering on those binary blobs to implement RF coexistence between BLE and another RF system, since none of the documented APIs support any of the necessary features.
FreeRTOS is open source and part of the code included in ESP-IDF. The controller is a task that runs under FreeRTOS. Though I think there is a mode where one dedicates a CPU core to the controller, on those ESP32 devices with two cores, and FreeRTOS doesn't manage the controller.
Depending on the ESP32 device, the controller code is either in the mask ROM and/or as a binary library file shipped with ESP-IDF. The older ESP32 appears to have more in the ROM while the newer chips like the ESP32-C6 have less ROM and use more libraries.
There's a performance issue with code in flash memory (which a library compiled into the FW would be), since the ESP doesn't have built in flash on the processor's data bus. It's an external SPI flash chip in all cases, which means possible long latencies when a cache miss causes the flash controller to need to read a block over SPI. Not something a timing critical part of the BLE controller can tolerate. The way around this is to copy the code that can't tolerate flash cache misses from flash into RAM, but there is very little RAM, especially on the OG ESP32. So they add ROM that doesn't have a cache miss issues to hold some of the code. ROM is much cheaper than RAM in a microcontroller.
Bluedroid and Nimble talk to the binary blob running in FreeRTOS. That's the controller for them- not the hardware.
No, bluedroid or nimble run as a task under the open source FreeRTOS kernel. They talk to the binary blob using a virtual HCI interface. It's not even a binary blob. They ship .a libraries that don't even have symbols stripped for the non-open-source code.
We'll see wtf is going on here, but it's very strange that the HCI command table includes these encodings.
The HCI spec includes an OGF for vendor specific commands, 0x3f, which is what Espressif has used. I fail to see what is strange about vendor specific commands using an opcode set aside in the spec for purpose of adding said commands.
It's perfectly possible that HCI EVTs coming from the actual hardware controller can also trigger these commands.
If the controller has some kind of RF triggered backdoor, there would be zero reason to create an HCI command to implement it. The controller code would just do the secret thing when triggered. You wouldn't add a customer facing command called "TriggerSecretBackdoor", which has no purpose other than to let customers activate the backdoor, if you wanted to add a backdoor that was secret from your customers.