Engineering a CI-Triage Decision System

The on-call engineer can stop the release, isolate the test, rerun the job, or investigate manually. A model cannot choose among those actions responsibly until the team writes down what its mistakes cost. That is why this build begins with the operational decision rather than a model.

The system estimates. The engineer acts.

A new component should not appear because an algorithm sounds suitable. It should appear because the current system cannot answer a concrete question. We will call these “observer questions”: the specific uncertainty a component is meant to reduce. Here is the seven-step loop used throughout.

The rest of this article follows that loop. Expand a build notebook to inspect the project reasoning. File chips identify local artifact paths; they are provenance references, not public download links. The main explanation stands on its own.

Before choosing a model, ask: what decision is a person trying to make, and which part of that decision can the system safely inform?

The system reads the current CI report and a test's run history. It emits a probability that the failure is flaky. It may refuse when evidence is thin. It does not merge, quarantine, delete, or stop anything.

This boundary matters because a probability and an action have different owners. The model can be technically correct about uncertainty while the release policy still chooses to stop. Conversely, the model can be confident and the engineer can override it because the incident carries context the model does not have.

Why the four verdicts need more than a binary classifier

1 − P(flaky) is a probability of “not flaky” under the model’s label definition. It is not automatically P(real defect). Non-flaky cases can include infrastructure faults, deterministic test defects, and incomplete evidence. ABSTAIN is a decision to withhold a verdict, not a fourth physical cause. UNKNOWN at ingestion and a model timeout are different again.

The lab combines a run-level gate, binary observers, and a human boundary. It has not trained a validated four-cause incident classifier. That gap matters: never let the product UI give a binary model a broader meaning than its training labels support.

The external contract contains four verdicts: REAL_DEFECT, FLAKY, INFRA, and ABSTAIN. The fourth exists because a first-time failure with no logs and no history does not fit safely into the first three.

These are estimates, not audited finance figures. Writing them down still improves the system because the team can challenge a visible assumption. An unwritten cost is harder to inspect and easier for an accuracy metric to erase.

ABSTAIN is cheap relative to shipping a defect, but it is not free. If every case is escalated, model risk falls while the system stops doing useful work. Coverage therefore has to sit beside risk.

Derive the decision threshold instead of inheriting 0.5

For this derivation only, assume two mutually exclusive incident states: flaky and real defect. Assume correct decisions have zero incremental cost, missed defects cost Cmiss, and unnecessary holds cost Chold. This deliberately simplified model is not the full four-verdict system. It also assumes p estimates the incident state, a stronger claim than the project’s historical flakiness label supports.

With $5,000 and $500, the threshold is 0.9091. At p = 0.8, dismissal has expected loss $1,000 and a hold has expected loss $400. The more likely class is flaky, yet a hold has lower expected loss. The costs explain why. On an equality, keep the more cautious action in this illustration. Threshold selection is separate from fitting a score; the scikit-learn threshold guide makes the same distinction.

The general rule is a*(x) = argmina Σy P(y | x)L(a, y): for each allowed action, add its losses across possible states, weighted by their probabilities, and choose the smallest. This supports three causes and a review action once the probabilities and loss table exist. It does not invent the missing probabilities for you.

The evaluator accepts probabilities and labels. It does not know whether the probabilities came from a tree, a GRU, retrieval, or an LLM. Every observer passes through the same function.

# Condensed artifact interface; imports and helper bodies omitted.
# Stored cost weighting is historical, as explained above.
def evaluate(probs, labels, threshold=0.5):
    probs, labels = _check(probs, labels)
    preds = (probs >= threshold).astype(int)
    return {
        "accuracy": accuracy_score(labels, preds),
        "precision": precision_score(labels, preds, zero_division=0),
        "recall": recall_score(labels, preds, zero_division=0),
        "roc_auc": roc_auc_score(labels, probs) if len(set(labels)) > 1 else None,
        "brier": brier_score_loss(labels, probs),
        "ece_equal_width": _ece(probs, labels, "equal_width"),
        "ece_equal_freq": _ece(probs, labels, "equal_freq"),
        "cost_weighted_risk": _cost_weighted_risk(preds, labels),
        "risk_coverage": _risk_coverage(probs, labels, threshold),
    }

The first measurement experiment used an always-not-flaky predictor against a roughly 3 percent positive rate. It achieved more than 95 percent accuracy and exactly zero recall. A second experiment tried to expose a calibration binning trap, failed to construct it, then corrected the probability distribution and confirmed it. This pattern becomes the unit of work for the rest of the build: state the expected behavior, construct a case, run it, and keep falsification visible.

Read a score by reconstructing its denominator

With 1 = flaky, precision is TP/(TP+FP): among flagged tests, how many carry a flaky label? Recall is TP/(TP+FN): among flaky-labeled tests, how many did we flag? ROC AUC measures ranking across positive-negative pairs. None directly prices a release mistake. The operational two-state loss would be (5,000 FP + 500 FN)/N, the reverse of the stored lab weighting.

Calibration asks whether cases assigned a probability near 0.8 are positive about 80% of the time. Fit the model on training data, fit calibration using independent held-out predictions, choose the decision policy on validation data, and inspect the final frozen policy on an untouched test set. For small data, use a properly nested cross-validation design. The calibration documentation describes both calibration curves and the need to separate calibration data from model fitting.

Coverage has two denominators

Execution coverage is completed requests divided by submitted requests. Decision coverage is issued verdicts divided by submitted requests. A completed request may abstain. Report both, plus the conditional error rate on issued verdicts and the total cost across all submitted requests.

A risk-coverage curve that drops refused cases from its denominator describes the accepted subset. It is useful, but it does not show total service cost. Nor is distance from 0.5 necessarily decision confidence when the action threshold is 0.9091. Sort by a validated risk or decision-margin estimate appropriate to the policy.

A test is labeled flaky when identical code produces both a pass and a failure across reruns:

The rule matched all 26,765 rows in test_results.csv. Yet the two classes do not have equal certainty. An observed flip establishes nondeterminism under the recorded conditions; infrastructure and uncontrolled environment changes still need investigation. A test that never flipped only proves that no flip was observed within its rerun budget.

This resembles a positive-unlabeled problem: some apparent negatives may be undiscovered positives. If observed positives are correct and errors only hide positives among negatives, measured precision on the same cases is a lower bound on true precision. That is a conditional statement. Infrastructure contamination can violate the first assumption, and selective rerunning can make the sample unrepresentative. Record rerun counts; do not turn this observation into an unconditional production guarantee.

How much can ten clean reruns tell you?

Assume each rerun independently fails with constant probability q. The chance of seeing no failures in n reruns is (1 − q)n. For a 1% failure probability and ten reruns, that is about 90.4%. A flaky test can look completely stable in a short observation window.

The independence assumption is strong. A shared machine outage can correlate outcomes; test order can change the per-run failure probability. Record commit, environment, seed, order, and run identity where possible. “Same code” controls only one source of variation.

There is also selection bias: engineers may rerun suspicious tests more often. Then num_reruns partly measures human attention. A model can learn the investigation policy rather than the behavior of the software. Use rerun budget as label provenance; admitting it as a predictive feature requires proving it exists at the intended decision time and does not encode the answer.

When two datasets must be joined, write the ways a row can disappear, duplicate, mismatch, or leak the answer before writing the join.

The most convenient dataset, IDoFT, had no detected license in the checked repository. Public access did not grant reuse rights. The build chose the FlakeFlagger dataset from Zenodo under CC BY 4.0 and kept attribution as part of the design.

The join looked simple until it returned zero rows. One file used names such as apache-commons-exec; the other used commons-exec. Normalizing test identity and matching the project suffix repaired the join. The finished table contained 26,142 rows across 25 projects with a positive rate of 0.0316.

key = normalized_project + "::" + normalized_class + "::" + normalized_method

assert joined_rows are within the expected source bounds
assert no feature is near-perfectly correlated with label
assert no feature's missingness reveals label

The last check found a real leak: flaky_source revealed the answer through missingness. It was non-null exactly where the label was positive. The team also removed history-derived modification windows that could not be recreated at prediction time from the available snapshots.

Freeze the world at the moment of prediction

Imagine the failing build arrives Tuesday at 02:47. Wednesday’s fix commit, Friday’s incident label, and a retrieval neighbor added next month are unavailable on Tuesday. A retrospective join can silently import all three. Every feature needs an event timestamp, an availability timestamp, and a rule that says why it was knowable at inference.

Store later outcomes in the label table. A test’s identity is not enough to guarantee a safe join: a feature with the right ID but the wrong time can still leak. Normalize identifiers, assert key uniqueness before joining, report unmatched rows by project and class, and inspect any suffix matching for collisions. Expected row counts alone cannot detect a wrong one-to-one match.

A failed build can make many tests look broken even when they never ran. The trust gate reads maven.log and emits one verdict with a numerical reason. Precedence is fixed: BUILD_FAILED, LOG_TRUNCATED, MASS_FAILURE, TRUSTED, then UNKNOWN.

The 5 percent threshold came from a real run with 13 failures out of 163 tests, about 8 percent, sharing SSL and query-parameter failure shapes. A proposed 15 percent threshold would have admitted that run, so it was lowered.

The gate later exposed a deeper lesson. The first parser read only the final Maven summary in a multi-module build. Another parser missed inline <<< ERROR! failure lines. These bugs silently removed failures from square-okhttp and distorted the sequence, retrieval, fusion, and distillation phases downstream.

A gate changes the population you claim to understand

Let G = 1 mean the gate admitted a run. Downstream metrics then describe P(error | G = 1). They do not describe all CI traffic. A strict gate can improve that score while refusing most difficult incidents. Report admission rates by project and failure type, review samples of rejected runs, and retain rejection reasons.

The 5% mass-failure rule is a heuristic motivated by an observed case. Many failures can also arise from a legitimate shared-code regression. Treat the gate’s diagnosis as evidence requiring inspection, not proof of infrastructure causation. A parser correction must invalidate affected histories, retrieval indexes, and evaluations together.

A random split mixed tests from the same project across train and test. A grouped split held out whole projects. The model, rows, and features stayed fixed; only the question changed.

The 0.206 gap is consistent with the model relying on project-specific patterns; it does not by itself isolate identity leakage from other distribution shifts. The grouped result also varied sharply depending on which project was held out. A single average would have hidden that instability.

At this point the team faced a deployment decision. Should one global model work cold on a new codebase, or should each team train a local instance on its own history? The first requires grouped evaluation. The second permits within-project evaluation but creates cold-start and model-maintenance costs. The system eventually chose the second shape and kept both results visible.

For a team’s future incidents, use a chronological boundary and build every historical aggregate and retrieval index as of that boundary. For unseen tests within a familiar project, separate test identities. For a new customer repository, hold out projects. These are different targets; choose the one implied by deployment and use additional splits as stress tests.

A strong evaluation records the target population, independent unit, time cutoff, allowed inputs, baseline, metric, and decision threshold before fitting. Keep feature selection, calibration, threshold tuning, and learned fusion inside the training/validation process. A final test set stops being a final test set once it repeatedly chooses the next design.

HistGradientBoostingClassifier reads 16 leak-checked numeric features. It handles missing values internally. The repository does not implement the earlier idea of per-row abstention for missing features, and the design document was corrected to say so.

Sigmoid calibration improved ECE from 0.0312 to 0.0225 but slightly reduced AUC. The original hypothesis said ranking would remain identical. It did not, because CalibratedClassifierCV(cv=3) trained three smaller base models and averaged their outputs. It did more than apply one monotone transform to a fixed score vector.

The 0.88 figure is not an improved global model. It comes from random folds within each project, evaluated on unseen tests from a known project. Nineteen projects had enough positives; six were skipped. This pivot trades cold-start generalization for local history.

What the numeric observer actually learns

A tabular row turns a test into numbers such as execution duration or coverage statistics. A decision tree partitions that feature space with questions of the form “is duration greater than this value?” Gradient boosting adds trees to improve the current predictions. It can discover interactions without you enumerating every combination.

That strength can also hide shortcuts. “Long runtime” may identify one repository’s tooling rather than a general cause of flakiness. Feature importance describes how this fitted model uses its inputs; it does not establish that changing a feature would change flakiness. Compare against a simple baseline, inspect errors by project, and remove suspicious features in an ablation before telling a causal story.

Choose a sequence model only after you can name what order contains that an aggregate destroys.

A naive sequence model would read the same pass/fail history used to define IsFlaky. It could learn the label rule directly: spot at least one pass and one failure. The build avoided that circularity by asking the prefix to predict whether the future suffix would flip.

Only one original project had enough trusted, nameable failures to build positive sequence examples. The small GRU achieved raw AUCs around 0.43 to 0.54. A free control, failure count in the prefix, scored roughly 0.52 to 0.58 and beat the GRU in nearly every fold.

The keep rule had been written before training: the GRU had to beat the control by at least 0.05 without collapsed calibrated probabilities. It failed. The team discarded it and retained the heuristic. More epochs or a larger hidden state were rejected because 23 distinct tests did not contain enough diversity for capacity tuning to repair.

What would make order worth modeling?

Consider P P P F F F and F P F P F P. Both fail three times in six runs. The first might suggest a change point; the second might suggest alternation. A count-based feature cannot distinguish them. A recurrent model maintains a hidden state: hₜ = update(hₜ₋₁, outcomeₜ), then predicts from that state. A GRU uses learned gates to control what it retains and replaces.

The model only earns its complexity if those ordering patterns predict a held-out future target better than counts do. Keep the prefix and future suffix separate; never derive an input statistic from the suffix you are trying to predict. Randomize order as a diagnostic: if performance is unchanged, the learned system may not need sequence structure. Discarding this observer after it lost to its simple control is a valid engineering result.

Retrieval embeds a failure message, finds nearby historical cases, deduplicates neighbors by test identity, and returns the fraction labeled flaky. It refuses to build a single-class index and abstains when it cannot find enough distinct tests.

query = embed(failure_text)
neighbors = nearest(index, query)
neighbors = dedupe_by_test_identity(neighbors)

if len(neighbors) < min_distinct:
    return ABSTAIN

p_flaky = mean(neighbor.label for neighbor in neighbors[:k])

The first corpus used real exception text for failures and synthetic “test passed” sentences for the other class. That created a possible style shortcut. The corrected square-okhttp corpus used real trusted exception text for both classes: 102 cross-run deterministic tests and 76 tests labeled flaky by the ground-truth table.

The pre-registered prediction expected performance to fall when the style cue disappeared. It did not. That prediction was falsified. The result is still within-project. A separate historical 17-project reference scored 0.474 against a 0.715 majority baseline, so the article cannot claim cross-project retrieval works.

Similarity is a representation choice, not a causal explanation

An embedding maps text to a numeric vector. Cosine similarity compares vector direction. Nearest-neighbor search retrieves cases close under that representation; it does not know whether the exception actually has the same cause. Library names, boilerplate, and duplicated stack frames can dominate similarity.

The vote 3/5 = 0.6 is a local class fraction, not automatically a calibrated incident probability. Distinct test identities prevent literal duplicate votes, but five tests from one shared helper can still carry one correlated failure. Inspect neighbor distance, shared code, label provenance, and age. Minimum neighbor count alone does not ensure relevant neighbors.

@dataclass(frozen=True)
class EvidenceRecord:
    observer: str
    case_id: str
    probability: float
    calibrated: bool
    has_evidence: bool
    read: str  # the checked-in contract uses a source-description string
    cost_dollars: float
    latency_ms: float

The constructor rejects probabilities outside [0,1], negative cost, negative latency, and an empty read declaration. Absence of evidence is carried explicitly rather than hidden as a neutral-looking probability. All downstream strategies receive exactly three records, one per observer.

A richer LLM prompt might include the raw log, test name, or retrieval neighbors. This experiment deliberately withheld them. Otherwise a better LLM result could come from having more information rather than from better fusion.

How one fusion case is assembled

build_fusion_cases() begins with the Phase 4 feature table and its test identifiers. It holds out square-okhttp for tabular scoring, loads deterministic-test exclusions and cached trusted runs once, and builds the corrected real-text retrieval corpus from that same cache. It then joins the tabular and retrieval populations by test identity.

The label join is defensive. If the Phase 4 label and corrected retrieval label disagree for the same test, construction raises instead of choosing one silently. Only shared, aligned tests enter evaluation. The tabular model is fit on every other project. Sequence evidence uses only the first 30 trusted outcomes. Retrieval leaves the query test out of its own index before finding neighbors.

shared_ids = align_or_raise(phase04_labels, retrieval_labels)
tabular.fit(other_projects)

for test_id in shared_ids:
    tabular_p = tabular.predict(test_features)
    sequence_p = mean(first_30_trusted_outcomes)
    retrieval_p = vote(index_without(test_id), k=5)
    emit(three_evidence_records)

The embedding model is loaded once. Corpus text and all query text are embedded in batches. The earlier implementation loaded the sentence-transformer inside the per-case path, which caused a 120-second smoke run to time out. Batching fixed the execution path without changing the experiment population.

When sequence or retrieval has no evidence, its record carries has_evidence=False and a placeholder probability of 0.5. Every strategy filters on has_evidence. This detail prevents the placeholder from becoming a real vote.

Count independent failure domains, not the number of boxes in the diagram.

The tabular observer reads the aggregate feature CSV. Sequence and retrieval both depend on raw archives and the trust gate. They may use different representations, but a parser bug can move both together. Two agreeing numbers can therefore be one upstream mistake repeated twice.

INDEPENDENT_GROUPS = [
    {"tabular"},
    {"sequence", "retrieval"},
]

An override requires agreement across at least two groups. This is the same principle used in reliable services: replication does not protect against a shared dependency failure.

Why counting votes can count a bug three times

Suppose two sensors each have a 10% error rate. If their errors were independent, both would be wrong on 1% of cases. If a shared parser makes them fail together, their joint error could be 10%. Knowing their individual error rates does not tell you which world you are in.

Dependency groups are a practical warning system, not a mathematical proof of independent errors. Measure paired mistakes on the same held-out cases. If you learn a combiner, train it on out-of-fold observer predictions rather than their fitted training predictions. Compare against the best individual observer, remove each observer in turn, and keep the population fixed across comparisons.

Absolute probability distance is the lab’s disagreement heuristic. It measures numerical separation, not threshold crossing by itself; a decision-aware policy must also check which side of its actual cost-derived threshold each score falls on. The case 0.49 versus 0.59 matters operationally even if a logit-space measure would describe 0.89 versus 0.99 differently.

The LLM sees only the records and must return JSON with a probability and one-sentence reason. Its probability is marked uncalibrated. A 30-second timeout is an execution error. It is not an abstention, because the model never completed a decision.

Why timeout handling reaches the process group

The LLM strategy invokes a command-line client as a subprocess. A normal child-process timeout can kill the direct child while one of its descendants still owns the standard-output pipe. The parent then waits for an end-of-file marker that never arrives. The implementation starts the client in a new session and kills the complete process group on timeout.

proc = subprocess.Popen(
    ["claude", "-p", prompt, "--output-format", "text"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
    start_new_session=True,
)

try:
    stdout, _ = proc.communicate(timeout=30)
except subprocess.TimeoutExpired:
    os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
    proc.communicate()  # reap the process group
    raise TimeoutError("arbiter timed out")

A dedicated test replaces the process with a hanging stub and asserts that killpg receives the process-group identifier and SIGKILL. Another test forces an arbiter failure and checks that the report increments n_errors while leaving n_abstained at zero.

Before comparing scores, compare denominators, coverage, execution errors, cost, and latency.

The corrected square-okhttp export aligned 178 cases and passed the predeclared variation gate. A stratified sample of 20 cases, roughly balanced by ground truth, went through all five strategies.

The two LLM rows cannot be ranked against the complete local rows. Their scored subsets may be easier, harder, or simply different. Twelve escalation calls and ten all-LLM calls timed out. Mean latency for the partial LLM rows was measured in seconds, while local strategies were effectively immediate.

A historical precomputed run over 74 cases showed best-single, weighted, and stacking strategies at 0.905 accuracy. A naive LLM arbiter scored 0.432; a calibrated version improved to 0.784 but still trailed the local methods. That reference is useful context, but it is not the corrected own run.

The valid fusion conclusion is about architecture and failure handling. The corrected experiment does not establish a five-way winner.

The teacher prompt receives real failure text and the three evidence records. Ground truth is withheld. The teacher emits a structured label and explanation. The operator-facing output pairs prose with the raw records so a fluent sentence cannot replace its evidence.

This division also limits what a failed student can damage. If the student collapses to one class, the decision layer still exists. Explanation quality can be evaluated, replaced, or removed without changing the triage verdict.

Separate label learning from explanation quality

The pilot asks the teacher to supply both a label and prose. The cheap TF-IDF gate measures label prediction. It does not measure whether an explanation is accurate, complete, useful, or faithful to the fixed fusion verdict. A future student would need a separate explanation evaluation; the current artifact has no such student result.

Start with a deterministic template: “Retrieval found three distinct historical tests; tabular evidence disagrees; the system abstained.” An LLM must improve usefulness beyond that baseline without adding unsupported causes. Require each factual clause to reference an evidence ID. Count unsupported claims, contradictions with the verdict, omitted disagreement, and valid refusals on an independently reviewed sample.

Logs and retrieved text are untrusted input. They may contain secrets or sentences that look like instructions. Redact sensitive material before model calls, delimit evidence, deny tool authority to the explanation component, validate structured output, and check evidence references. An instruction in a stack trace must never become an instruction to the triage service. JSON validity alone cannot establish factual grounding.

The project estimated a 3 billion parameter model in 16-bit precision. The point was not to predict one exact peak-memory number. It was to identify which technique attacks which term.

LoRA freezes the base model and trains small adapters. It removes base-model gradients and optimizer states from the training burden, but it does not make the frozen weights disappear. Quantization reduces the weight term by storing the base model at lower precision. Activation checkpointing reduces stored activations by recomputing them, trading compute for memory.

A rough 4-bit base is about 1.5 GB before quantization metadata and runtime buffers. The project estimated a QLoRA-style job might fit within a 24 GB GPU, but that estimate was never validated by an actual training run because the data gate failed first.

The exact teacher message shape

You are labelling training data for a CI triage explanation model.
Given the real failure evidence and detector evidence below, write a grounded
2-4 sentence explanation for an on-call engineer. Decide whether the test is
FLAKY or NOT_FLAKY. Do not invent facts absent from the evidence.
State confidence between 0 and 1.

Test: {test_fqn}
Failure evidence:
{failure_text}

Detector evidence:
{evidence}

Respond with ONLY this JSON object:
{"explanation": "...", "verdict": "FLAKY" or "NOT_FLAKY",
 "confidence": <float 0-1>}

The true label is used to balance and evaluate the pilot but is not inserted into this message. The teacher must reason from the same failure and detector evidence that would exist at inference time.

The initial estimate treated the prompt as the billed request and predicted roughly $0.001 per call. One smoke call cost $0.0174. The CLI carried system prompts, tool definitions, project context, cache operations, and thinking overhead that the rendered user prompt did not show.

The corrected pilot sampled 60 of 178 valid square-okhttp cases, balanced 30/30 by hidden ground truth. All 60 teacher calls parsed. A group-disjoint split produced 48 training rows and 12 test rows. The student gate was intentionally cheap: TF-IDF over unigrams and bigrams with a class-balanced logistic regression.

The rule said TF-IDF had to beat the majority baseline before any GPU work. It tied. The stop condition fired. No pod was rented, no LoRA job ran, and no student result exists for the corrected corpus.

A separate historical fine-tune reference had collapsed to predicting FLAKY on every example. It matched a majority baseline on one majority-FLAKY slice and scored zero on a deterministic slice. This is evidence about that historical run, not a substitute result for the corrected pipeline.

The corpus pipeline has its own safety boundary

Importing phase12_prepare.py does not call a teacher or start a GPU job. Running it without --run-teacher validates the source cases and exits. The paid path is therefore an explicit command-line action rather than an accidental side effect of a test or import.

# Read and validate the Phase 11 export
rows = load_cases("phase11-cases.json")

# Choose 30 + 30 using hidden ground truth
pilot = balanced_cases(rows, limit=60, seed=0)

# Paid operation, allowed only by an explicit flag and budget
labelled, spend = label_with_teacher(
    pilot,
    model="claude-haiku-4-5-20251001",
    budget_usd=2.0,
)

# Keep test identities disjoint
train, test = balanced_group_split(labelled, test_fraction=0.2)

# Gate before GPU work
result = run_baseline(train, test)

The loader requires a case identifier, ground-truth label, real failure text, and Phase 11 observer records. It rejects conflicting labels for a repeated identity and rejects a corpus that does not contain both classes. The teacher never receives the hidden label. It sees the test identity, real failure evidence, and normalized observer lines containing probability, calibration state, and evidence availability.

The response parser accepts only FLAKY or NOT_FLAKY, checks confidence is between zero and one, requires an explanation, and preserves the provider's actual cost field. This last field matters because token estimates had already failed. The batch loop stops admitting calls when accumulated spend reaches the budget. This is a soft admission limit: the last admitted call can take the total over budget. A hard cap would need a conservative reservation for each call and a provider-side limit where available.

Why the split follows test identity

Repeated examples from one test could share package names, exception text, and detector patterns. If one landed in train and another in test, the baseline could memorize the test family. balanced_group_split() holds the complete test_fqn on one side and asserts that both labels remain in both partitions.

The final JSONL files retain prompts, evidence, teacher label, confidence, explanation, token counts, and cost. They are audit records as much as training data. If the student later produces a suspicious explanation, the team can trace it to the teacher example rather than guessing what the model absorbed.

The collaboration used a repeated loop:

The AI also made mistakes. It proposed redundant metrics, invented an attribution before checking the real record, wrote docs that claimed unimplemented behavior, undercounted teacher cost, and initially treated a timeout as abstention. These errors became useful only because the workflow preserved design records, experiment predictions, result artifacts, and rejected proposals.

The most important lesson in the repository appeared late. The Maven parser dropped legitimate failures in a multi-module project. Four phases then produced separate-looking conclusions from the same damaged evidence.

After the parser repair, square-okhttp produced a two-class real-text retrieval corpus, a fusion population with variation, and a teacher pilot with both classes. The sequence observer still failed its own control, but the other three conclusions changed materially.

This is why an AI-native build needs provenance beyond a final notebook. A result must retain which parser, gate, corpus, split, and execution path produced it. When an upstream assumption changes, the team needs to know which downstream claims expire together.

Read the project in dependency order rather than filename order:

1. PROBLEM.md for the incident, boundary, and cost assumptions.
2. design/00 through design/13 for slice responsibilities and explicit gaps.
3. decisions/ for the choices that constrain implementation.
4. experiments/ for predictions written before results.
5. ci_triage/ and tests/ for the actual mechanism and invariants.
6. artifacts/results/ for executed outputs, including partial coverage.
7. ai-ledger/, KNOWNS.md, and FAILURES.md for rejected proposals, corrections, and handoff risks.

The core dependency chain is:

A practical rerun should begin with the cheap invariants and local strategies. Paid teacher calls and GPU work need separate budget gates. The checked-out artifact used Python 3.12 dependencies including pandas, scikit-learn, PyTorch, sentence-transformers, pytest, and the Anthropic client. The result files are the evidence for the numbers in this article. They should not be rewritten to make corrected code and historical output look artificially consistent.

The finished artifact is not one model. It is a chain of contracts that makes weak evidence, failed experiments, partial execution, and human judgment visible.