Closed Loop, Open For Feedback?

We’ve come far as a country, from role-playing Hollywood-level combat scenarios in Atropia to embedding artificial intelligence in virtual warfare simulations. The U.S. military leaves no stone unturned in preparing its soldiers for the uncertainties of war. But if “closing the loop” is what makes the system better, who remains outside to evaluate it?

Shield AI’s recent acquisition of Aechelon Technology paints a picture of a future where, as Shield AI CEO Gary Steele remarked in a statement, “the future of warfare will be defined by humans and autonomous systems operating and fighting together as a networked team.”

Shield AI is a defense technology company that develops AI-powered pilots, drones, and intelligence software for U.S. military operations. Hivemind, its renowned autonomous software, has been used to build and scale AI pilots, whether by leading operations in communication-jammed environments or coordinating multi-agent swarms. The company has contracts with the U.S. Department of Defense and the Department of Homeland Security.

Aechelon Technology is also a defense technology company, but one that specializes in ultra-realistic synthetic realities, 3D visualizations, and sensor simulation software for the U.S. armed forces and allied militaries. It has provided technology and software to the U.S. Special Operations Command to equip soldiers with more realistic mission rehearsals. Its software powers flight simulations, conducts mission rehearsals, and trains human and autonomous AI pilot systems.

Together, their collaboration would allow for deeper integration between Aechelon’s virtual simulation technology and Hivemind’s autonomous software. Gary Steele reaffirmed in the acquisition press release that “every action, by human and machine, must be tested and validated in simulation and fed back in a continuous, closed data loop.”

Shield AI is quick to reject the looming “black box” label that often undermines the legibility of AI models. Its Chief Technology Officer, Nathan Michael, has said that the company built Hivemind specifically so third parties can develop their own engineering skills instead of being locked into a “closed black box.” In a customer story about the credibility of Shield AI’s workflow, the company admits that, before building stronger pattern-tracing tools with the help of Foxglove, “without a clear way to understand why an aircraft chose a specific flight path, operators were left with black-box behaviors in high-stakes environments.”

Like Hivemind, Shield AI’s investment in Aechelon Technology will involve manned-unmanned teaming. The developer builds it, the customer deploys it, and the system’s closed data loop tests it. The software’s behavior will be logged, traceable, and reviewable. But who is that traceability for? The Foxglove tools embedded in Hivemind are for staff engineers to assess system precision and debug. Despite being a customer story, there is no customer quote or outside auditor’s perspective in that account. Just because a system is now more traceable to its engineers and developers does not mean it is more accountable to everyone else.

The gap between developer and customer transparency could widen with the Aechelon deal. Before the acquisition, Aechelon was an outside vendor selling visual simulation software to the U.S. military. In a company blog post published after the deal closed, Aechelon co-founder Nacho Sanz-Pastor, now Shield AI’s General Manager of Aechelon, described the two companies as previously “solving the same problem from different directions.” Now, Shield AI builds the autonomous systems, owns the simulator that tests them, and then reports and improves upon its own results.

So, the problem is not that the technology is unreliable. Shield AI and Aechelon Technology credibly market their products as tested for safety and efficiency. The question, rather, is whether company-led validation is enough to produce legitimate confidence in artificial intelligence products. You can prove that a system is reliable, but it is your company that designs the system. Your company builds the test. And it is your company that evaluates the outcome.

And then there is the question of whether the test could catch an issue that it was not originally designed to detect.