Serious props to the reporters on this story (they are not active on Bluesky so I'm not tagging them). We are all genuinely lucky that news illustrating what can go wrong when militaries rely on hallucination-prone LLMs got published before something worse happened.
- A US special operations command analyst used an AI chatbot to assess a Chinese ship's manifest, and it falsely reported nuclear weapons components.
- Armed troops were preparing to board the vessel and planes were airborne before officials caught the error and halted the operation.
- The chatbot fused open-source and classified signals intelligence, and the analyst then used AI again to package the report for wide dissemination.