Methodology
Version m1.0. Each task gets a tier, a stability mark and a timeline. Each tier says exactly what was found and where.
The tiers
| Tier | Meaning | How it is produced |
|---|---|---|
| Tier A | The issue or discussion states the change that fixes the problem: replacement code, a diff, or a sentence naming the exact edit. | Model verdict, with every quote checked as a word-for-word substring of the text. |
| Tier B | The text names the exact file, function or line to change, without giving the change. | Model verdict with the same quote check. Shown separately from Tier A. |
| Tier C | Added lines of the gold patch appear in the issue or discussion. | Deterministic string match after whitespace normalisation. No model. |
| Links the fix PR | The issue text or the discussion links the pull request whose change is the gold patch. | Deterministic: a link to the instance’s own pull-request number, found by string match. Shown as its own signal; it does not change the tier. |
| No quoted evidence | Nothing in the text states or points to the fix. | A finding about the text only, not proof that the task is clean. |
Quotes cannot be invented
The model returns passages it believes carry the evidence. Each is located in the stored issue or discussion text by exact match. A passage that is not found word for word is dropped, and a task left without a verified quote is shown as having no quoted evidence.
Two independent checks
Every task is judged twice, with differently worded instructions and different reasoning effort. When both reach the same tier the task is marked Both runs agree. When they differ it is marked Runs differ and left out of the headline counts. If one of the two found a quote, the task is shown as Disputed with that quote, so it is never read as clear. Tier C is a string match, so this mark never applies to it. Both runs and their rationales are on the task page.
Git timeline
For each task Leaklens locates the upstream commit that made the same change, using the repository’s public history, and shows its date and link. This is a date, not a verdict: it says public exposure was possible from that day, so you can compare it with any model’s training cutoff.
Wording
Leaklens reports what the public text and history contain. It never says a model was trained on a task or memorised it; it says the fix was in the issue, or the fix commit was public on a date.
Data and licence
Task records come from the public SWE-bench datasets (SWE-bench repository, MIT licence; issue text belongs to its GitHub authors). Leaklens displays the text only to quote evidence, and links back to the source pull request.
Cite a result
Leaklens, SWE-bench Verified task <instance_id>, methodology m1.0. https://leaklens.grain64.com/t/<instance_id>
See how far to trust the tiers on the validation page. Think a flag is wrong? Report it.