Biosecurity review needs the selected system, not a random output

A random sample of a model’s outputs can be the wrong object to evaluate when another system selects which outputs matter.

That is the governance question I take from recent genome-design research. Samuel King and colleagues’ Science paper, published August 6, reports experimental work on bacteriophages—viruses that infect bacteria. The finding should not be retold as proof that a model can generally design dangerous human pathogens.

A June reanalysis by James Black and colleagues separates evolutionary novelty from design efficiency. It emphasizes the combined contribution of the model and additional filtering, and limits the generality of its conclusions beyond the systems studied. It is a preprint, which should remain visible when its conclusions are cited.

My proposed review boundary is the full system that selects an output and gives it a path toward action. A model-only score can be useful. It cannot describe a capability that emerges from the model, selection process, expert curation and downstream access working together.

Average output and operational capability can diverge

An evaluation can ask how often a model produces a result of interest from a randomly chosen request. A deployed workflow may instead generate several candidates and pass only a selected subset onward. Those are different populations.

The selection process can make rare useful results more likely to reach a decision-maker. It can also filter out problematic outputs. Whether it amplifies capability, reduces risk or does both depends on the specific system. The review needs evidence about that system rather than an assumption that downstream processing is neutral.

This is not an argument for publishing operational biological methods. It is an argument for making the evaluation boundary explicit to the qualified people responsible for oversight. The relevant review can remain at the level of capability, controls and access without turning a public report into a reproduction guide.

I would want the assessment to identify which components were included, which were omitted and which conclusions depend on the omission. A result measured before selection should not be described as the result of the final workflow.

Keep novelty and efficiency on separate axes

The distinction in Black and colleagues’ analysis is particularly useful. A system may become more efficient at finding viable results near known examples without demonstrating reliable invention far beyond them. Increased efficiency can matter even when novelty is limited.

The reverse is also important for interpretation. An unusual generated artifact is not automatically functional, and a functional artifact is not automatically evidence of a wholly new capability. A dramatic headline can collapse those questions into a single claim that the research does not support.

I would keep the two axes separate in a governance report. State what evidence supports a claim about efficiency, what supports a claim about novelty and what remains untested. Do not allow the strength of one result to fill a gap in the other.

For biological work, judgments about hazard and appropriate controls require relevant domain expertise. A generic agent benchmark cannot supply that expertise. The useful contribution from engineering is to make the evaluated configuration and decision path inspectable so specialists can assess the actual system.

Review the handoffs that change the consequence

Here is a hypothetical institutional review process. A team proposes a research workflow using a generative model and downstream selection. I would ask the team to present a high-level map of each handoff, the responsible owner and the evidence required before progressing.

The map would distinguish computational evaluation from any authorized physical work. It would identify where expert review occurs and what stops progression when the evidence or permission is insufficient. It would not treat model access as permission for every later stage.

rendering diagram…

This diagram is a review structure, not an experimental protocol. Its purpose is to prevent a change in capability or access from disappearing between organizational owners.

If the model remains unchanged but the selection process changes, the reviewed system has changed. If access to a downstream service expands, the consequences may change even without a new benchmark score. Both deserve an explicit decision about whether the prior assessment still applies.

Carry the lesson into ordinary enterprise evaluation

The same measurement problem appears in lower-stakes business systems. Consider a hypothetical assistant that drafts many responses but sends only the one selected by a ranking stage. Testing random raw drafts would not tell me what customers actually receive.

I would evaluate the selected responses and the conditions under which the ranking stage fails. I would also test the permission boundary before delivery. That analogy is about system evaluation, not an equivalence between customer-service errors and biological risk.

The practical artifact is a versioned description of the whole evaluated workflow: model, relevant selection components, access conditions and human decisions. Keep the results attached to that description so a later component change cannot inherit approval by name alone.

Research becomes more useful when the limits remain intact. We can take the governance lesson seriously without overstating the novelty of the biology or understating the importance of selection.

Evaluate the system that chooses and can act on outputs; a model-only sample cannot certify the workflow around it.