Repository navigation
Add Spanner GoogleSQL corpus and parse its DDL gaps - #1
Merged
Merged
Conversation
Add cmd/spanner-corpus, which scrapes the SQL examples from Cloud Spanner's GoogleSQL reference pages (DDL, DML, query, pipe, procedural, graph and function pages), runs each through the reference parser with Spanner's language features, and writes accepted examples as golden cases under parser/testdata/spanner/. Examples the reference rejects (Spanner-only syntax, doc typos, grammar fragments) are listed in per-page .rejected.sql files for triage. The parser tests and next-test now include the Spanner files. 1529 of 1545 cases pass; the 16 remaining todos are HIDDEN columns, PROPERTY GRAPH DDL and DROP VECTOR INDEX. Move the execute_query download/run logic into internal/refparse so regenerate-parse and spanner-corpus share it. Pass "--" before the SQL (a leading "--" comment was taken as a flag), detect errors printed to stdout with a zero exit status, and pass --output_asc_explicitly to match the parser test driver's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UYsqdZSDA9cbKQdQissXaR
Scrape the GoogleSQL examples from ~130 Spanner guide pages (found with the new -discover mode) in addition to the reference pages; their cases go under parser/testdata/spanner/guides/. PostgreSQL tabs are skipped and query-result comments are stripped (some are closed with a typo'd "/*" that swallowed the rest of the block). Rejected examples are now classified instead of dumped: - Grammar templates, partial clauses, bare graph patterns, "-- Error:" examples and similar are recognized by heuristics and dropped. - Examples that list statements without ";" are split and retried. - The rest are classified by id in cmd/spanner-corpus/triage.txt as spanner (written to <page>.spanner.sql, the Spanner-only syntax backlog), typo or fragment. Untriaged rejects go to <page>.rejected.sql; there are none now. 2476 cases; the 87 that fail are all PROPERTY GRAPH DDL, HIDDEN columns or DROP VECTOR INDEX. 160 Spanner-only statements are in the backlog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UYsqdZSDA9cbKQdQissXaR
- HIDDEN column attribute (HiddenColumnAttribute) in column_attributes.
- CREATE [OR REPLACE] [scope] PROPERTY GRAPH with node/edge element
tables: KEY, SOURCE/DESTINATION KEY ... REFERENCES, LABEL / DEFAULT
LABEL, NO PROPERTIES / PROPERTIES [ARE] ALL COLUMNS [EXCEPT (...)] /
PROPERTIES (derived, ...), DYNAMIC LABEL / DYNAMIC PROPERTIES, and
element and statement OPTIONS. DROP PROPERTY GRAPH as a generic
DropStatement.
- DROP {SEARCH|VECTOR} INDEX [IF EXISTS] name [ON table].
- ALTER {SEARCH|VECTOR} INDEX with SET OPTIONS, ADD/DROP COLUMN,
ALTER COLUMN SET OPTIONS and REBUILD actions.
Trees and error messages were checked against the reference parser,
including the error paths. This clears the last Spanner corpus todos;
no parse_todo cases remain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UYsqdZSDA9cbKQdQissXaR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a test corpus built from the SQL examples in Cloud Spanner's GoogleSQL docs, and teaches the parser the syntax that corpus showed was missing.
Spanner corpus (
cmd/spanner-corpus)execute_query, withSPANNER_LEGACY_DDLand the graph features enabled). Accepted examples are written as golden cases underparser/testdata/spanner/(2476 cases);TestParserandnext-testnow pick them up recursively.-- Error:and bare graph patterns (-vlists them).;between them are split and retried.cmd/spanner-corpus/triage.txt. Spanner-only syntax that the reference parser rejects is written to<page>.spanner.sql; this backlog has 160 statements (GRANT … TO ROLE, change streams, proto bundles, locality groups, identity columns, …). Doc typos are dropped. Untriaged rejects go to<page>.rejected.sql; there are none now.-discovercrawls the docs navigation for guide pages that aren't inpages.goyet.execute_queryhandling moved intointernal/refparse, shared withregenerate-parse. Three bugs fixed along the way:--is no longer read as a command-line flag.execute_queryprints to stdout with exit code 0 are now treated as errors.--output_asc_explicitlyis passed, matching the test driver's output.Parser
HIDDENcolumn attribute.CREATE [OR REPLACE] [scope] PROPERTY GRAPHwith node and edge element tables:KEY,SOURCE/DESTINATION KEY … REFERENCES,LABEL/DEFAULT LABEL, thePROPERTIESforms,DYNAMIC LABEL/PROPERTIES, andOPTIONS. AlsoDROP PROPERTY GRAPH.DROP {SEARCH|VECTOR} INDEXandALTER {SEARCH|VECTOR} INDEX(SET OPTIONS,ADD/DROP COLUMN,ALTER COLUMN SET OPTIONS,REBUILD).Parse trees and error messages were checked against the reference parser, including about 60 malformed inputs. No
parse_todocases remain.Test plan
go test ./... -timeout 60spasses.go test ./parser -check-parseleaves no todos.go run ./cmd/spanner-corpusregenerates identical output.🤖 Generated with Claude Code
https://claude.ai/code/session_01UYsqdZSDA9cbKQdQissXaR
Generated by Claude Code