Commit 23227fd
authored
Add query routing and execution handoff for Parquet-backed indices (#5267)
* [Mustang] Add query routing and execution handoff for Parquet-backed indices (#5247)
Implements the query routing and AnalyticsExecutionEngine for Project
Mustang's unified query pipeline. PPL queries targeting parquet_ prefixed
indices are routed through UnifiedQueryPlanner and executed via a stub
QueryPlanExecutor, with results formatted through the existing JDBC
response pipeline.
New files:
- QueryPlanExecutor: @FunctionalInterface contract for analytics engine
- AnalyticsExecutionEngine: converts Iterable<Object[]> to QueryResponse
with type mapping and query size limit enforcement
- RestUnifiedQueryAction: orchestrates schema building, planning,
execution on sql-worker thread pool, with client/server error
classification and metrics
- StubQueryPlanExecutor: canned data for parquet_logs and parquet_metrics
tables for development and testing
Modified files:
- RestPPLQueryAction: routing branch for parquet_ indices
- SQLPlugin: passes ClusterService and NodeClient to RestPPLQueryAction
- plugin/build.gradle: adds :api dependency for UnifiedQueryPlanner
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Align isClientError with RestPPLQueryAction classification
Add missing exception types to isClientError(): IndexNotFoundException,
ExpressionEvaluationException, QueryEngineException,
DataSourceClientException, IllegalAccessException. Matches the full
list in RestPPLQueryAction.isClientError().
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Move stub code into analytics/stub sub-package
Extract StubSchemaProvider, StubQueryPlanExecutor, and StubIndexDetector
into plugin/.../rest/analytics/stub/ package to clearly separate
temporary stub code from production code. RestUnifiedQueryAction now
delegates to these stub classes.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Use ErrorMessageFactory for error responses
Replace hand-crafted JSON error response with
ErrorMessageFactory.createErrorMessage(), matching the standard error
format used in RestPPLQueryAction.reportError().
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Make metrics and response formatting query-type-aware
Use QueryType to select the correct metrics (PPL_REQ_TOTAL vs REQ_TOTAL,
PPL_FAILED_REQ_COUNT_* vs FAILED_REQ_COUNT_*) and LangSpec (PPL_SPEC
vs SQL_SPEC) so this class can serve both PPL and SQL queries when
unified SQL support is added.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Move analytics routing from REST layer to transport layer
Move the analytics index routing check from RestPPLQueryAction into
TransportPPLQueryAction.doExecute(). This ensures the analytics path
gets the same PPL enabled check, metrics, request ID, and inter-plugin
transport support as the existing Lucene path. RestPPLQueryAction and
SQLPlugin are reverted to their original state.
Added executeViaTransport() to RestUnifiedQueryAction which returns
results via ActionListener<TransportPPLQueryResponse> instead of
RestChannel, integrating properly with the transport action pattern.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Add query size limit to RelNode plan instead of post-processing
Add LogicalSystemLimit to the RelNode plan before passing it to the
analytics engine, consistent with PPL V3 (QueryService.convertToCalcitePlan).
This ensures the analytics engine enforces the limit during execution
rather than returning all rows for post-processing truncation.
Remove post-processing querySizeLimit truncation from
AnalyticsExecutionEngine -- the limit is now part of the plan the
executor receives.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Propagate security context to sql-worker thread pool
Wrap scheduled lambdas in both execute() and executeViaTransport() with
withCurrentContext() to capture and restore ThreadContext (user identity,
permissions, audit trail) on the worker thread. Follows the same pattern
as OpenSearchQueryManager.withCurrentContext().
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Move analytics routing after profiling setup
Move the analytics index routing check after QueryContext.setProfile()
and wrapWithProfilingClear(listener). Use clearingListener instead of
raw listener so profiling thread-local state is properly cleaned up
after analytics path execution.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Remove NPE from isClientError classification
NPE in the analytics path is a server bug (null schema field, missing
row), not bad user input. Removed from client error list. Will sync
this classification with RestPPLQueryAction updates in #5266.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* Remove duplicate REST execution path from RestUnifiedQueryAction
Remove execute(query, queryType, channel), doExecute(), createQueryListener(channel),
recordSuccessMetric(), recordFailureMetric(), reportError(), and related
REST imports. Since routing now goes through TransportPPLQueryAction,
the REST-specific path was unused. Renamed executeViaTransport() to
execute() as the sole entry point.
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>
---------
Signed-off-by: Kai Huang <kaihuang@amazon.com>
Signed-off-by: Kai Huang <ahkcs@amazon.com>1 parent 513e1b2 commit 23227fd
11 files changed
Lines changed: 1119 additions & 1 deletion
File tree
- core/src
- main/java/org/opensearch/sql/executor/analytics
- test/java/org/opensearch/sql/executor/analytics
- integ-test/src/test/java/org/opensearch/sql/ppl
- plugin
- src
- main/java/org/opensearch/sql/plugin
- rest
- analytics/stub
- transport
- test/java/org/opensearch/sql/plugin/rest
Lines changed: 110 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
Lines changed: 32 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
0 commit comments