Introduce new query option to execute the query using FDB's snapshot isolation - #4364
Introduce new query option to execute the query using FDB's snapshot isolation#4364ScottDugas wants to merge 47 commits into
Conversation
0de34c0 to
dbb2fe3
Compare
This is maybe more appropriate as documentation for options in general, rather than snapshot specific
I believe the jdbc serialization is only actually concerned with connection options, but all options use Options.Name regardless of where they are usable, so the tests make sure that any can be set across JDBC. It's probably best to support all, and let the connection reject the option.
Treat ISOLATION LEVEL SNAPSHOT like the other query options at connection scope: setting it on a connection applies it to that connection's SELECTs, letting a user set it, run a few reads, and switch back. No production change is needed -- a connection-scoped option already flows through to the query -- so this just adds a test in OptionScopeTest confirming a SELECT works on a connection that sets the option. (Mutations on such a connection still hit the existing SELECT-only check, exactly as a per-query snapshot option would.)
dbb2fe3 to
3043b16
Compare
And cleanup the MAX_EVER example
| The ``ISOLATION LEVEL SNAPSHOT`` query option runs a ``SELECT`` at FoundationDB's **snapshot | ||
| isolation** instead of the default serializable isolation. Under snapshot isolation, the reads |
There was a problem hiding this comment.
We’re talking specifically about what they call Snapshot reads, right?
We could link to that informative ACID section or the Snapshot Reads section in the FoundationDB Developer Guide. Though such an external link might not age well.
There was a problem hiding this comment.
Yeah, I thought about linking to that. I was a little concerned that it might be more confusing than beneficial. Particularly of note is that FDB only has read-write conflicts, but a lot of things in SQL add reads, so I could see it being confusing.
I think a more informative ACID section would be beneficial, but I think it might be better to do that as a followup, rather than bringing into this PR.
| Terminal('NOCACHE'), | ||
| Sequence(Terminal('LOG'), Terminal('QUERY')), | ||
| Sequence(Terminal('DRY'), Terminal('RUN')), | ||
| Sequence(Terminal('PLAN'), Terminal('RIGHT'), Terminal('DEEP')), | ||
| Sequence(Terminal('ISOLATION'), Terminal('LEVEL'), Terminal('SNAPSHOT')) |
There was a problem hiding this comment.
Not sure how “official” those other options like PLAN RIGHT DEEP are. Unless we’re also describing them somewhere, I’d vote to rather just leave them out for now.
There was a problem hiding this comment.
We don't have a separate staging for "official" features, the documentation tries to align as much as possible with the current state of the code (that's why we have documentation tests).
If something is "too internal" we should mention that in the documentation that a particular language feature is subject to change in non-backward-compatible way, but that's pretty much it.
There was a problem hiding this comment.
Nit: I’m trying to steer us to remove hard line breaks from .rst and .md files, so writers don’t have to reflow paragraphs they edit all the time, and readers can rely on soft wrapping to get their preferred line width. Many pages already do this, though not all of them yet.
There was a problem hiding this comment.
Sounds fair. I've definitely been annoyed having to rewrap text, but mostly that has been javadoc.
There was a problem hiding this comment.
I did that as a separate commit, so hopefully you can easily review the wording changes independently, and that's helpful
| An option affects only the statement it is attached to. It does not change any connection-wide or | ||
| transaction-wide setting, and it is not remembered for later statements — each statement that needs | ||
| an option must specify it. |
There was a problem hiding this comment.
Does the remark in ISOLATION_LEVEL_SNAPSHOT.rst about having to re-supply the same option to EXECUTE CONTINUATION apply in general, or is it something specific to ISOLATION LEVEL SNAPSHOT? If yes, we should mention it on this page as well. If not, we could at least hint at there also being such a thing as a “per-execution option”.
There was a problem hiding this comment.
That's a good point, the same reasoning would apply to the other runtime option DRY RUN that @pengpeng-lu added a while ago which probably follows the same principles documented here .
There was a problem hiding this comment.
I think it applies to DRY RUN and LOG QUERY, although I don't know that we have tests of either combined with continuations. Given that we don't have documentation for either of those (Adding it seems like an independent body of work outside this PR), and we don't have a lot of other options yet, it seems better to keep the continuation characteristics at a lower level to avoid locking us into "Options are not continued".
DRY RUN may actually be worse to not include in the continuation, but TBH it kind of seems like for that we should include it in the continuation, and require the same value as an option to ensure that we are doing what the developer expected.
But, I think the bottom line for this PR is that I think we should leave the behavior and documentation as-is but maybe think about how it could be better going forward, especially as more options are added.
| // correct. Executing at SNAPSHOT isolation would, at a minimum, require adding any records read to the conflict | ||
| // range to ensure index consistency. It also requires determining and documenting the exact semantics. Because | ||
| // of this complexity and a lack of immediate requests, this is not supported. | ||
| if (executeProperties.getIsolationLevel().isSnapshot()) { |
There was a problem hiding this comment.
To match the comment, rather:
| if (executeProperties.getIsolationLevel().isSnapshot()) { | |
| if (executeProperties.getIsolationLevel() != IsolationLevel.SERIALIZABLE) { |
Also in RecordQueryDeletePlan.java.
There was a problem hiding this comment.
Looking at that isSnapshot boolean inside IsolationLevel, I find that really silly, because it’s just a redundant way to distinguish IsolationLevel.SNAPSHOT. Could get rid of that, but I see it existed already way before this PR.
There was a problem hiding this comment.
I changed this in the helper method. There is definitely a style around the code outside this change of having is* methods like this, but I'm not sure why.
|
|
||
| The ``ISOLATION LEVEL SNAPSHOT`` query option runs a ``SELECT`` at FoundationDB's **snapshot | ||
| isolation** instead of the default serializable isolation. Under snapshot isolation, the reads | ||
| performed by the query do **not** add read-conflict ranges to the enclosing transaction, so |
There was a problem hiding this comment.
Regarding “the reads performed by the query do not add read-conflict ranges”, this is specifically about the various plan nodes that access the data, i.e., the scans. We need to be sure that all the different types of index scans indeed honor that isolation level in ExecuteProperties. If not all of them actually do that yet, it’d be good to add a caveat to this statement.
There was a problem hiding this comment.
Yeah, we do, and I do have a test of all the indexes, but I wanted to build that into more of a framework, and decouple it from merging this work. The tests added here do confirm record scans, and a couple indexes, but are not exhaustive, and keeping them exhaustive would be hard. You can see a very early draft of that framework: #4414
| * This only affects the scans of the query it is set on; other reads and writes in the same transaction | ||
| * continue to use their configured (typically serializable) isolation. Only supported on read-only | ||
| * ({@code SELECT}) queries. | ||
| * Scope: Query |
There was a problem hiding this comment.
Actually Scope: Connection, Query, right? According to that snapshotIsolationTakenFromConnection test.
There was a problem hiding this comment.
Yeah. TBH I wouldn't have added the connection version at all, but some tests or code required it, or did it implicitly.
| Assert.that(isSelect || isExecuteContinuation, | ||
| ErrorCode.UNSUPPORTED_OPERATION, | ||
| "OPTIONS (ISOLATION LEVEL SNAPSHOT) is only supported on SELECT queries"); |
There was a problem hiding this comment.
But aren’t there also IS_EXECUTE_CONTINUATION_STATEMENT statements that aren’t SELECT statements, like COPY? Those would slip through here.
There was a problem hiding this comment.
Yes, that is a good catch. I have made some updates to restrict it to just continuing on select statements. It may add to the argument to whether we want to reconsider how options relate to continuations.
| ``ISOLATION LEVEL SNAPSHOT`` is a **per-execution** option: it is applied to the execution it is | ||
| specified on and is **not** stored in the continuation. When a query is paginated and resumed with |
There was a problem hiding this comment.
That’s a bit of a footgun. Would it be hard to implement such an automatic propagation into the continuation?
There was a problem hiding this comment.
I think I would be more concerned about going the other way. Accidentally running something at snapshot isolation when you expect it to be serializable would result in application-level inconsistencies, but accidentally running at serializable means increased chance of conflicts.
Most of the other options that we have don't make sense across continuations, but the ones that do (dry run and log query) are not carried across continuations. maxRows is also per request, explicitly.
You could argue that having to respecify those other items is also bad, but I think we should assess holistically rather than making the isolation level special.
| * Snapshot isolation changes only conflict detection, not visibility. A snapshot read still returns | ||
| data as of the transaction's read version and never sees uncommitted or later-committed writes | ||
| from other transactions. It will include writes from the current transaction. |
There was a problem hiding this comment.
This is an important point, not just a restriction. I.e., that a snapshot read really observes the same “snapshot” as a serializable read; no actual difference in that regard between the two. It’s worth working that into the Overview above already.
There was a problem hiding this comment.
Yeah, that's a good point. Everything in this bullet except RYW is already in the overview. I tried to expand there in a natural way, driving home all the key points. And removed this bullet point entirely
I left tests, and some booleans alone, for readability.
hatyo
left a comment
There was a problem hiding this comment.
LGTM, provided addressing Robert's comments.
…Children I tried to reuse a planGraph helper in a test, and this failed because it couldn't cast the insert plan to an update plan. Looking at FoundationDB@1668ad3#diff-962c865223da499cc697b5060170f1543c0cef6558b2b9388cc8187a4392ea71 I think this is was a mistake extracting the base class
I used idea, so this should be pretty trustworthy
This gives us better forward compatibility
- capitalize sql terms - remove pointless queries - comment on purpose of each query
Conflicts on some Options related helpers because someone else added an option too
ScottDugas
left a comment
There was a problem hiding this comment.
I think I addressed everything. Some things I would like to defer.
| size;some;space;sql;sqlcode;sqlerror;sqlstate;struct;substring;sum;system_user;table;template;temporary;then; | ||
| timezone_;timezone_minute;to;trailing;transaction;translate;translation;trim;type;union;unique;unknown;update; | ||
| upper;usage;user;using;value;values;varying;view;when;whenever;where;with;work;write;year;zone" ignore_case="true" /> | ||
| size;snapshot;some;space;sql;sqlcode;sqlerror;sqlstate;struct;substring;sum;system_user; |
There was a problem hiding this comment.
isolation and level already existed.
| The ``ISOLATION LEVEL SNAPSHOT`` query option runs a ``SELECT`` at FoundationDB's **snapshot | ||
| isolation** instead of the default serializable isolation. Under snapshot isolation, the reads |
There was a problem hiding this comment.
Yeah, I thought about linking to that. I was a little concerned that it might be more confusing than beneficial. Particularly of note is that FDB only has read-write conflicts, but a lot of things in SQL add reads, so I could see it being confusing.
I think a more informative ACID section would be beneficial, but I think it might be better to do that as a followup, rather than bringing into this PR.
There was a problem hiding this comment.
Sounds fair. I've definitely been annoyed having to rewrap text, but mostly that has been javadoc.
| // Like other data-modification plans, a delete must run at serializable isolation: it reads existing records | ||
| // (to maintain indexes, etc.) and those reads must participate in conflict detection to remain | ||
| // correct. Executing at SNAPSHOT isolation would, at a minimum, require adding any records read to the conflict | ||
| // range to ensure index consistency. It also requires determining and documenting the exact semantics. Because | ||
| // of this complexity and a lack of immediate requests, this is not supported. | ||
| if (executeProperties.getIsolationLevel().isSnapshot()) { | ||
| throw new RecordCoreArgumentException("Cannot execute a data-modification plan at SNAPSHOT isolation level") | ||
| .addLogInfo(LogMessageKeys.PLAN, getClass().getSimpleName()); | ||
| } |
There was a problem hiding this comment.
Fair, at the time, I didn't feel it needed de-duplication for the 3 lines of code, but extracted it to QueryPlanUtils.
There was a problem hiding this comment.
I think it is worthwhile to have this defense-in-depth, to make sure that as things get combined you can't end up with a statement that sneaks it in. There is a test for connection options calling an insert, and seeing that it fails.
The delete & update tests do have a fair amount of boiler plate. I think it would be worthwhile to invest some effort into making it easier to construct plans more broadly, but that is outside the scope of this PR; I will take a second look to see if that can be a bit cleaner.
| private static final long SCAN_UPPER_BOUND = 1000L; | ||
|
|
||
| /** The query transaction runs; a full range scan over the data. */ | ||
| private static final String SCAN_QUERY = "SELECT id FROM t WHERE id < " + SCAN_UPPER_BOUND; |
There was a problem hiding this comment.
I may not be following you here, but there is no index defined on the id field, so this is not a covering index scan, its a record scan.
| try (ResultSet rs = statement.executeQuery(SELECT_QUERY)) { | ||
| ResultSetAssert.assertThat((RelationalResultSet) rs).hasNextRow().isRowExactly(0L); | ||
| } |
There was a problem hiding this comment.
Yeah, I was probably a bit lazy here, we should be able to test that the isolation level is applied.
At this point I think this PR is coming in with enough changes that I would prefer to go refactoring these tests, especially given how simple they all are
| - | ||
| - query: delete from t1 where id IN (select id from t1 where id = 1 OPTIONS (ISOLATION LEVEL SNAPSHOT)) | ||
| - error: "42601" # SYNTAX_ERROR | ||
| - | ||
| - query: insert into t2 (select id, 'name' from t1 where id = 1 OPTIONS (ISOLATION LEVEL SNAPSHOT)) | ||
| - error: "42601" # SYNTAX_ERROR | ||
| - | ||
| - query: create schema template SNAPSHOT_TEMPLATE create table tt(id bigint, v bigint, primary key(id)) create index bad_idx as select v from tt OPTIONS (ISOLATION LEVEL SNAPSHOT) | ||
| - error: "42601" # SYNTAX_ERROR | ||
| - | ||
| - query: select id from (select id from t1 where id = 1 OPTIONS (ISOLATION LEVEL SNAPSHOT)) as sub | ||
| - error: "42601" # SYNTAX_ERROR | ||
| - | ||
| - query: select id from t1 where exists (select id from t1 where id = 1 OPTIONS (ISOLATION LEVEL SNAPSHOT)) | ||
| - error: "42601" # SYNTAX_ERROR | ||
| ... |
There was a problem hiding this comment.
They aren't failing because of the specific option. We could add a separate options test, but I think these tests are valuable to provide context in case someone thinks about reverting #4362 or extend where the options can go. As with the other queries I added comments
| - | ||
| - query: select col1, max_ever(col2) from t1 group by col1 OPTIONS (ISOLATION LEVEL SNAPSHOT) | ||
| - unorderedResult: [{!l 10, !l 7}, {!l 20, !l 8}] | ||
| - | ||
| - query: select sum(col2) from t1 OPTIONS (ISOLATION LEVEL SNAPSHOT) | ||
| - result: [{!l 25}] | ||
| - | ||
| - query: select col1, sum(col2) from t1 group by col1 having sum(col2) > 12 OPTIONS (ISOLATION LEVEL SNAPSHOT) | ||
| - unorderedResult: [{!l 10, !l 15}] | ||
| - | ||
| - query: select max_ever(col2) from t1 OPTIONS (ISOLATION LEVEL SNAPSHOT, LOG QUERY) | ||
| - result: [{!l 8}] |
There was a problem hiding this comment.
A bunch of these are probably pointless now. so I removed some and added comments for the rest.
| Assert.that(isSelect || isExecuteContinuation, | ||
| ErrorCode.UNSUPPORTED_OPERATION, | ||
| "OPTIONS (ISOLATION LEVEL SNAPSHOT) is only supported on SELECT queries"); |
There was a problem hiding this comment.
Yes, that is a good catch. I have made some updates to restrict it to just continuing on select statements. It may add to the argument to whether we want to reconsider how options relate to continuations.
I added a method to fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/plans/QueryPlanUtils.java which was removed on main because the only method that existed on main was removed. Thus this class is added back, with just the new method
Add
OPTIONS (ISOLATION LEVEL SNAPSHOT)query optionSummary
Adds a per-query SQL option,
OPTIONS (ISOLATION LEVEL SNAPSHOT), that runs aSELECTat FoundationDB's snapshot isolation instead of the default serializable isolation. Under snapshot isolation the reads performed by the query do not add read-conflict ranges to the enclosing transaction, so concurrent writes to the data the query reads will not cause the transaction to fail when it commits.SELECT ... OPTIONS (ISOLATION LEVEL SNAPSHOT)The Record Layer core already supports
IsolationLevel.SNAPSHOTend-to-end (ExecuteProperties→ScanProperties→FDBRecordContext.readTransaction(snapshot)); this change exposes it at the SQL layer.Motivation / background
By default, every read in a transaction is serializable: FoundationDB records the range of keys that was read, and if any of those keys is modified by another transaction that commits first, this transaction is rejected with a conflict and must be retried. This guarantees a consistent view and safe writes, but it also means a read over a wide or frequently updated range can cause conflicts even when the exact values read do not matter.
A snapshot read still observes a consistent, point-in-time view of the database (as of the transaction's read version), but it does not register a read-conflict range. This makes snapshot isolation useful when a query reads data that is likely to be written concurrently and the query does not need its read to participate in conflict detection.
Snapshot isolation applies only to the reads of the statement it is attached to. Other reads and writes in the same transaction — and everything else on the connection — continue to use their normal (serializable) isolation, so a snapshot
SELECTcan be freely mixed with serializable reads and writes in a single transaction.Examples
Approximate row limit protected by a count index
Suppose a table has a
COUNT(*)index and the application wants to cap it at roughly a maximum number of rows:Before inserting a new row, read the current count and proceed only if it is under the cap:
Every insert into
documentupdates the singledocument_countindex entry. If the count were read at serializable isolation, that read would conflict with every concurrent insert, effectively serializing all inserts and causing frequent retries. Reading it at snapshot isolation adds no conflict range, so concurrent inserts proceed. The trade-off is that the limit becomes approximate: under high concurrency a few rows may slip in past the cap, because each transaction decides against a count that does not reflect the others' not-yet-committed inserts. This is usually acceptable for a soft limit.Sequence-like ids from a
MAX_EVERindex and a random offsetSuppose the application needs to assign roughly-increasing ids without a central sequence generator. A
MAX_EVERindex tracks the largest id ever assigned:To assign a new key, read the current maximum:
The application then adds a small random offset to
max_keyand inserts the row with that key (for exampleINSERT INTO folder VALUES (max_key + <random 1..100>, 'the-name')). Every insert updates the singlemax_folder_idindex entry, so — as in the previous example — reading it at serializable isolation would conflict with every concurrent id assignment. Reading it at snapshot isolation avoids that conflict, and the random offset makes it unlikely that two concurrent assignments choose the same key.Because snapshot reads do not conflict, two transactions can read the same maximum and act on it independently, so design for the possibility that another transaction derived the same value. Here that possibility is handled for free: if two transactions do pick the same new key, the primary-key write itself conflicts and one transaction retries. Widening the random range lowers the collision probability, at the cost of leaving larger gaps between assigned keys.
Restrictions
SELECT) statements (and, when resuming one, onEXECUTE CONTINUATION). It may be written on anINSERT,UPDATE, orDELETEstatement, but is rejected there because mutations rely on serializable reads (for example, when maintaining indexes and enforcing primary-key uniqueness) to remain correct. This is enforced both at the SQL layer (UNSUPPORTED_OPERATION) and, as a backstop, at the record-layer plan level (data-modification plans throw if executed at snapshot isolation). Allowing snapshot isolation for mutations is possible, but the exact semantics would need to be flushed out, and thus is beyond the scope of this change.SELECTs (a mutation on such a connection still hits the SELECT-only check above).Continuations
ISOLATION LEVEL SNAPSHOTis a per-execution option: it is applied to the execution it is specified on and is not stored in the continuation. When a query is paginated and resumed withEXECUTE CONTINUATION, the resumed execution runs at snapshot isolation only if the option is specified again on the resuming statement:If the option is omitted when resuming, the resumed pages fall back to the default (serializable) isolation and once again add read-conflict ranges — with no error or warning. To keep an entire paginated scan at snapshot isolation, repeat
OPTIONS (ISOLATION LEVEL SNAPSHOT)on everyEXECUTE CONTINUATIONcall.Implementation notes
ISOLATION LEVEL SNAPSHOTto the statement-level options rule (RelationalParser.g4); no new lexer tokens required.Options.Name.SNAPSHOT_ISOLATION(+ default/contract), a matchingStructuredQuery.QueryOptionsentry, parsing inAstNormalizer, and application inQueryPlan.executePhysicalPlan(overridesExecutePropertiesisolation for that execution only). The option is also carried over the JDBC/gRPC transport (jdbc.protofield +TypeConversion), so it works for remote connections, not just embedded.PlanGenerator; and a core-level invariant inRecordQueryAbstractDataModificationPlanandRecordQueryDeletePlanso no mutation can ever run at snapshot isolation, regardless of entry point.Testing
SnapshotIsolationConcurrencyTest(requires FDB): two transactions verifying the defining property — a snapshot read adds no conflict range so a concurrent write into the read range does not conflict, while the serializable control does. Covers plain scans, continuations (including the per-execution behavior on resume), read-your-writes, and joins/unions with concurrent writes to either input.OptionsTest/ProtobufConversionTest: option defaults, properties round-trip, and JDBC/gRPC serialization round-trip.snapshot-isolation.yamsql(functional coverage of query shapes + negative cases) andisolation-level-snapshot-documentation-queries.yamsql(keeps the doc examples runnable/correct).Documentation
Adds a new Statement options reference section describing the statement-level scope of
OPTIONS, and an ISOLATION LEVEL SNAPSHOT page (the source of the Overview/Examples/Restrictions/Continuations above).Closes: #4361