16Evidence
What has Cyber Warrior Network actually measured?
Little, and we say so. We separate 5 levels: vision, specified capability, implemented and reference-checked, sandbox-integrated, customer-validated. Nothing here is customer-validated. This page lists what sits at each level today, including a preregistered study that did not pass its gates, and what we have not measured.
Each result gives its date, method, what it does not show, and where it came from.
What has actually been measured
We separate 5 levels: vision, specified capability, implemented and reference-checked, sandbox-integrated, customer-validated. This table lists what sits at each level today.
| What | Level | Result | Scope and limits |
|---|---|---|---|
| Evidence-handling code (reference package) | Implemented and reference-checked | 1281 automated tests of the reference package pass (October 29, 2026) | Software tests of a reference library that no product feature uses yet. Not a performance result. |
| Scorer integrity check | Reference-checked on constructed test data | Passed once, at no spend, October 16, 2026 | Negative-control validation on test fixtures we constructed. It shows the checking pipeline behaves on cases we built ourselves, not that a real evaluator is sound. |
| Question: does a smaller, focused context view keep an AI worker's obligations as well as full context? | Preregistered study. Run October 17, 2026, 240 paired cases, small open model | Did not pass its preregistered gates. Verdict: do not promote. The smaller view missed required records less often, but it claimed completion falsely more often, beyond the tolerance set in advance. | Most model replies (about 92%) were in an unreadable format, so it is not a clean test of the idea. No uplift is claimed. |
Why we show a result that did not pass
The gates for the study were set before it ran. The smaller view did better on a first measure and worse on a second, and the second broke the tolerance we had set in advance. We publish that as it stands. A negative result shown honestly is more useful to a buyer than a positive result without its limits.
Because the holdout cases were used, a re-test needs a new preregistration with fresh cases, approved separately.
Where each number comes from
- Test count: a run of the reference package's own automated tests on October 29, 2026. The count changes as the code changes.
- Scorer integrity check: the recorded result of a single run on constructed test fixtures, checked against the plan that was frozen before it ran.
- Context study: the recorded result of the final run, checked against the plan that was frozen before it ran.
- The underlying result files are held by Cyber Warrior Network and are not published on this page. Treat these as our own reports, not as independent verification.
What we have not measured
- No customer uplift. No customer has run the first-hour comparison with us.
- No predictive result. The world-model research protocols (state, intent-conditioned context, forecasting, interactions, interventions, learning, transfer) are proposals. The context study above is the only study of that kind that has been run, and it did not pass. Apart from it and the scorer integrity check, no registered experiment has been run.
- No live savings. Replays are never reported as live savings.
- No result for mechanism or graph-representation research. A registered experiment of that kind has not been run.
- No insurance loss, premium or portfolio result. Our insurance data is public or constructed, and cannot be used to set premiums.
How to read a result from us
Every result will give its date, method, denominator, spend, what it does not show, and the evidence that produced it. If a test finds no benefit we will publish “no demonstrated uplift”.
What this page is not
- Contains no result from a buyer's own systems, and names no customer.
- Not a claim that any of this improves your outcomes. Your workflow is the next test.
- Not independent verification.
Quick answers
Has Cyber Warrior Network measured any customer uplift?
No. No customer has run the first-hour comparison with us, and this page does not claim an uplift. Everything on it is from our own lab, with its date, method and limits.
Did every study you ran pass?
No. A preregistered study of a smaller, focused context view did not pass its gates on October 17, 2026. The smaller view missed required records less often but claimed completion falsely more often, beyond the tolerance set in advance. Most replies from the small open model were unreadable, so it is not a clean test.
Why publish a result that did not pass?
Because a result shown with its limits is worth more than a result shown without them. If a test finds no benefit we will publish no demonstrated uplift.