Validation

Every number here comes from a labelled sample you can re-draw. Method, seed and sample sizes are on this page; each disagreement links to its task.

How the sample is drawn

Flagged tasks are sampled at random from each tier with a published seed; each sampled task is labelled by comparing its quoted text with the gold patch and recording whether the quote states or points to the change the patch makes. Each task was labelled in one pass, with no second reader, so the percentages measure agreement with that reading, not a consensus of several. A task’s own page shows its label and, for any disagree or unsure label, the reason. A separate random sample of unflagged tasks is labelled the same way to estimate what the detector misses.

How this relates to SWE-Bench+

The SWE-Bench+ study (arXiv 2410.06992) reports 37 of 500 Verified tasks with solution leakage under its own definition. Leaklens grades each task into an evidence tier instead of one yes/no, so the counts differ by design: a Leaklens Tier A means the text states the fix, whether or not a particular model benefited.

Cite it

Leaklens validation, methodology m1.0. https://leaklens.grain64.com/validation