Flywheel cycle one: lessons from 101 CPIDs

The first complete Library flywheel cycle ran end-to-end last week. Twenty-six sections from a single jurisdiction-specific corpus went through ingestion, atomisation, deduplication, and provisional staging. The numbers were narrow but instructive: 101 net new CPIDs, 23 atoms bypassed at the dedup boundary, and 392 atoms staged for review. Roughly two-thirds of the runtime was spent in the dedup and provenance phases — almost none in generation. That ratio is not a bug; it is the entire point of the architecture.

The zone-validation gate

The most consequential finding from the cycle was the failure rate of the zone-validation gate. Between 86 and 88 percent of slot assignments arrived at the gate marked invalid. Most failures concentrated in two zones — financial covenants and termination cascades — where atom boundaries straddle multiple canonical slots. In cycle one we treated the gate as a hard filter, dropping invalid slot proposals entirely. That cost the system real provenance: atoms that should have been retained as candidates were discarded with their source links.

In cycle two we will move to a softer gate. Invalid slots will be flagged but their atoms will be retained, routed instead to a triage queue where a reviewer disambiguates. The cost is a higher staging volume; the benefit is that no provenance is lost at the gate. Given that provenance loss is functionally irreversible — the source binding cannot be reconstructed from the atom alone — this is the correct trade.

Provenance persistence

The second architectural lesson concerns the ProvenanceBundle and its companion, the asymmetric containment scorer. The bundle survives every transformation in the pipeline: atomisation, dedup, slot assignment, validation, and staging. At no point is the bundle dropped or summarised; it accretes annotations as the atom moves. By the time an atom reaches staging, its bundle carries the full lineage from source document through to canonical slot proposal.

The containment scorer is what makes dedup safe at this lineage depth. A naive symmetric similarity score treats two atoms as duplicates when their content overlaps heavily. But contract atoms are not symmetric: a more specific atom contains a more general one, and the right behaviour is to retain the more specific atom and link the bundle of the general one to it. Asymmetric containment gives us that. It is also why our dedup retention rate (atoms kept versus atoms collapsed) is significantly higher than what symmetric methods would produce — the corpus is denser than it would otherwise be.

What changes for cycle two

Three concrete changes. First, the soft gate described above. Second, jurisdictional metadata propagated into the bundle from ingestion forward, rather than re-derived at slot assignment. Third, a staging-queue prioritisation pass that uses bundle density (number of source references per atom) to surface high-confidence atoms first. None of these is glamorous. All of them compound.