Skip to content

Commit 5930594

Browse files
mriggerclaude
mrigger
and
claude
committed
Add an /impact/ section: what SQLancer found, who uses it, who writes about it
The project's reach was folklore -- a bug count somebody remembered, a sense that papers cite it. This builds it as data instead: 2,130 records, each one carrying the source it came from and the reason it counts, collected by a pipeline that can be re-run and disagreed with. What the section says 2,038 bugs across 41 database systems, 541 of them found by people outside the lab. 22 projects whose own evidence shows them running SQLancer and 9 more that have proposed it. 209 external papers, 69 built on the tool or its techniques, 34 calling it state of the art. Nine recorded talks where somebody stood up and described it. The bug count is published as a floor, and the page says so. Closed-source systems report nothing, some open-source ones (Materialize) report elsewhere, and a bug report need not name the tool that found it -- six projects that demonstrably run SQLancer contribute no counted bug at all, and the page names all six rather than implying the gap away. How a claim gets in Every record cites a source and an attribution rule, and the rules are ranked: a project saying "found by SQLancer" outranks a technique name, which outranks the fact that a known reporter was running a campaign. The weakest rule alone admits nothing. Where prose is ambiguous an LLM classifies, under three constraints that shaped most of the design: it answers about text it was given and cannot fetch more; it never supplies a quotation, only cites the id of a sentence already extracted, which makes fabricated evidence structurally impossible rather than caught afterwards; and it may answer "uncertain", which routes the candidate to a person instead of publishing a guess. Answers are cached by content hash and committed, so the dataset rebuilds offline and a rerun costs nothing. Nothing is published on a name match alone. An acronym like TLP or CERT needs corroborating context; a reproducer that generates `t0(c0 ...)` is a fingerprint, but 556 of the records rest on it together with a known reporter, never on the shape by itself. Papers are read, not counted `_data/papers/` holds one file per paper: the sentences that mention SQLancer, the citation markers that resolve to it -- which is how a paper saying "existing works [9]-[11]" is found at all -- what its artifact contains, and the classification each sentence supports. This is what separates a paper that builds on SQLancer from one that runs it as a baseline, a distinction name-matching got backwards. The PDFs themselves stay out of the repository, permanently. Talks are read four ways Slides can be quoted, a frame from the recording can only be shown, captions are verbatim with respect to the caption track and nothing more, and a viewer's account is a note. The four are kept apart on the page, because an automatic caption is not a quotation and a picture of a slide is not text. Also here: a page per database system, per paper and per year of bugs; charts; a review queue for what no rule settled, down to nine items; JSON Schema over every data file; and 281 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N
1 parent 156b782 commit 5930594

1,240 files changed

Lines changed: 347892 additions & 110 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cache/impact/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Collector caches
2+
3+
Everything here is an optimisation. The pipeline produces the same records with
4+
this directory empty as with it warm, and a test asserts that.
5+
6+
What is committed is decided by one question: could a fresh checkout get this
7+
back by itself? A fetched page could, so it is not committed. An answer that
8+
cost tokens, a transcript a person had to read out of a browser, and the record
9+
of how far each source has been scanned could not.
10+
11+
| Path | Committed | What it holds |
12+
| --- | --- | --- |
13+
| `state.json` | yes | Last successful scan and last full reconciliation per source. This is what makes weekly runs incremental. |
14+
| `classifications/` | yes | Every classifier answer, including the negative and uncertain ones, keyed so that a change to the evidence, policy, taxonomy, prompt version or model forces a fresh answer and nothing else does. Only the answer and a hash of the evidence are stored, never the evidence itself. |
15+
| `talk-transcripts/` | yes | Windows of caption text around the moments a talk mentions SQLancer. Committed because a person had to obtain them by hand -- YouTube serves caption tracks only to its own player -- so no rerun can recover them. See that directory's README. |
16+
| `derived/` | yes | Deterministic extraction results, keyed by source id + content hash + extractor version. Empty so far: nothing has cost enough to keep. |
17+
| `http/`, `github/`, `papers/`, `artifacts/` | no | Fetched bodies with their ETag / Last-Modified / upstream `updated_at` validators. Large, re-fetchable, and restored from the GitHub Actions cache instead. |
18+
| `papers-pdf/`, `papers-text/` | no | Supplied PDFs and the text extracted from them. Deliberately never committed; only the two Markdown files explaining the directory are. |
19+
20+
Committing the classification cache is what keeps the weekly token cost
21+
proportional to genuinely new evidence: a paper that turned out to only cite
22+
SQLancer, or an issue that turned out to be unrelated, is never asked about
23+
again.

‎.cache/impact/classifications/.gitkeep‎

Whitespace-only changes.

‎.cache/impact/classifications/00/00ec0c4ab2395175ad39e0db6af27133acf0ece46c7310ab724e35aa023f3468.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:sourcenetwork/defradb.rs#157|sha256:0159711df6fd52cd2a699b6c60acd1bc8047472e69973af3824d6a7ef28c924c|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "Names SQLancer, but as work to be done rather than as the source of a defect: adoption, CI plumbing, a feature SQLancer needs, a release checklist or a roadmap. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:0159711df6fd52cd2a699b6c60acd1bc8047472e69973af3824d6a7ef28c924c",
19+
"source_id": "github:sourcenetwork/defradb.rs#157",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/02/022bc8ec1b7eaddf62d93509b5b4f2beb7ea6fdbca92818282945fd4f8592e5c.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:apache/datafusion#14535|sha256:9fceee6658a445ab1f52c3c607ce61416d06f32872dd4d16b38e5da1f3896984|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "Names SQLancer, but as work to be done rather than as the source of a defect: adoption, CI plumbing, a feature SQLancer needs, a release checklist or a roadmap. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:9fceee6658a445ab1f52c3c607ce61416d06f32872dd4d16b38e5da1f3896984",
19+
"source_id": "github:apache/datafusion#14535",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/02/0291964c195d72f8a24f4c54a20e0149e1eb1994ae783a10e6bd6b5f43dceb42.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:cockroachdb/cockroach#171420|sha256:a3c4e88181f416bcfcc47103f659f614cea733da51110147d481b561a1de04f0|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "\"cert\" here is a TLS certificate -- --ca-cert, cert-password, a cert file or a certificate warning -- not Cardinality Estimation Restriction Testing. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:a3c4e88181f416bcfcc47103f659f614cea733da51110147d481b561a1de04f0",
19+
"source_id": "github:cockroachdb/cockroach#171420",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/02/02b1ecf20f9657dd9f6fba8da840822cdcd50a35e7cb8c4656d70629bd6cdcaa.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"key": "github:yugabyte/yugabyte-db#10236|sha256:d009f54b290299815ca53cce87b5723188d8f2c320cae0c72c694d475c419ecf|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "yes",
11+
"excerpts": [
12+
"SQLancer: IOException occurs on UDPATE statement"
13+
],
14+
"extra": {
15+
"finder": "sqlancer",
16+
"technique": null
17+
},
18+
"reason": "The report attributes the find to SQLancer. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
19+
},
20+
"source_hash": "sha256:d009f54b290299815ca53cce87b5723188d8f2c320cae0c72c694d475c419ecf",
21+
"source_id": "github:yugabyte/yugabyte-db#10236",
22+
"taxonomy_version": "taxonomy-v2"
23+
}
24+
}

‎.cache/impact/classifications/03/03b1a6cffb6b046f97f2ef36d208258d7cc148dd2ef47da3f5d0c64a0482e69c.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:databendlabs/databend#9448|sha256:7dcd4cd0e809656806ecd0d617ff11284dd2f7b0cc9704140858aafaf661f416|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "Names SQLancer, but as work to be done rather than as the source of a defect: adoption, CI plumbing, a feature SQLancer needs, a release checklist or a roadmap. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:7dcd4cd0e809656806ecd0d617ff11284dd2f7b0cc9704140858aafaf661f416",
19+
"source_id": "github:databendlabs/databend#9448",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/05/05a9632477e1270abb608e2426c322ffe2afc56f509a58453dffa2bb78c8ec72.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:cockroachdb/cockroach#10517|sha256:9e8fa41de7c8985480545d2866fc42c35d16c68a9008682756189cd329c45ef2|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "\"cert\" here is a TLS certificate -- --ca-cert, cert-password, a cert file or a certificate warning -- not Cardinality Estimation Restriction Testing. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:9e8fa41de7c8985480545d2866fc42c35d16c68a9008682756189cd329c45ef2",
19+
"source_id": "github:cockroachdb/cockroach#10517",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/05/05e103a12eb6ae65021d96b2743d0a7a7d8367f6711135e566db2ca1957f7560.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:cockroachdb/cockroach#84484|sha256:137209a7e0f3d138ec5dd07b57203cc45d4c97a57dd67f856d396f16cd8daacf|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "The TLP roachtest failed for a reason that is not a finding: an expired licence, a cluster that would not start, a library that would not load, a timeout, or no stated cause at all. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:137209a7e0f3d138ec5dd07b57203cc45d4c97a57dd67f856d396f16cd8daacf",
19+
"source_id": "github:cockroachdb/cockroach#84484",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

‎.cache/impact/classifications/05/05f9a6064d39f03460452b8b9d3e0450c637f97ea01b4705276e5d80d80d0283.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"key": "github:cockroachdb/cockroach#144923|sha256:e736395c073f9f044c9d08687afc01f28a64ead7fa9871a03d3b34f4539e8bc3|bug-attribution-v1|impact-policy-v1|taxonomy-v2|claude-opus-5",
3+
"stored_at": "2026-09-13T03:09:23Z",
4+
"value": {
5+
"classified_at": "2026-09-13T03:09:23Z",
6+
"classifier_version": "bug-attribution-v1",
7+
"model": "claude-opus-5",
8+
"policy_version": "impact-policy-v1",
9+
"result": {
10+
"answer": "no",
11+
"excerpts": [],
12+
"extra": {
13+
"finder": null,
14+
"technique": null
15+
},
16+
"reason": "The TLP roachtest failed for a reason that is not a finding: an expired licence, a cluster that would not start, a library that would not load, a timeout, or no stated cause at all. Decided by reading the issue in this session rather than by an API call; the excerpt was checked to be literally present in the text the classifier would have been given."
17+
},
18+
"source_hash": "sha256:e736395c073f9f044c9d08687afc01f28a64ead7fa9871a03d3b34f4539e8bc3",
19+
"source_id": "github:cockroachdb/cockroach#144923",
20+
"taxonomy_version": "taxonomy-v2"
21+
}
22+
}

0 commit comments

Comments
 (0)