Add eight more engines: Presto, Hive, Kylin, XuguDB, Db2, Informix, SUNDB and GBase 8s - #59
Merged
Merged
Conversation
Eight engines join the CLI, four of them with a driver the release can ship and four that connect with a driver the user provides. Bundled through the JDBC worker, staged by scripts/jdbc-fixture.sh and validated by tests/databases.py: Presto, Hive (the last release whose driver still runs on the pinned JRE 17), Apache Kylin and XuguDB. The packaging script now refuses to ship a driver compiled for a newer Java than the pinned runtime, which is how the Hive version was chosen. Provided by the user, because their vendors do not allow redistribution: IBM Db2, IBM Informix, SUNDB and GBase 8s. `sqlx driver add --type <engine> --jar <path>` copies the vendor jar into <data-dir>/drivers/<engine>/ after checking it carries the engine's driver class, `sqlx driver list` reports where each engine's driver comes from, and `sqlx driver remove` clears it. A provided jar also wins over a released component, so a newer vendor driver can be used without waiting for a release. A statement for one of these engines without a driver is refused before anything is downloaded, naming the exact command to run. The JDBC worker gains a URL template per engine, the parameter block Informix and GBase 8s need for their server instance, and two fixes found by testing against real servers: a driver that does not implement getMoreResults no longer ends the statement with a spurious error (Hive), and a driver that renders its message lazily now does so while its own jar is still open, so Db2 reports the SQL error instead of dying with NoClassDefFoundError. Db2 also needs no FROM-less SELECT, so the shared one-row probe is per engine. Verification: cargo fmt/clippy/test, mvn -f java/jdbc/pom.xml verify, the UI build, the distribution guards, a full scripts/package-shared.py run, and tests/databases.py against live Presto, Hive and Db2 fixtures. Kylin, XuguDB, Informix, SUNDB and GBase 8s ship with fixtures, drivers, probes and skill pages but could not be exercised on this machine: the Kylin image does not start under arm64 emulation, the XuguDB trial image does not publish its SYSDBA password, the Informix developer image rejects the client host, the SUNDB image license expired in 2022, and the local GBase 8s instance was offline. Each of those is documented in tests/databases.py next to the environment variable that supplies it. Chat2DB gains the matching engine mapping for the eight types.
The secrets context is not available to a step condition, so the workflow file failed to parse. The four engines that need vendor assets stay documented in tests/databases.py and the README instead.
GBase 8s connects, runs DDL and DML and returns exact numeric values, but its driver never implements isValid or getMoreResults, and both are reported as a plain SQLException saying the method is not supported. A failed validation call is now inconclusive rather than fatal, and only a driver that says the method is unsupported ends the result loop early. Its URL also carries the client locale, without which the driver fails inside its own parser, and --property DB_LOCALE or CLIENT_LOCALE replaces the default. Informix and GBase 8s have no DROP TABLE IF EXISTS and no FROM-less SELECT, so the integration harness drops tolerantly and asks each engine for its own one-row probe. GBase 8s also has no usable VARCHAR with driver 3.70.1.61 (a stored value reads back empty while CHAR and LVARCHAR round-trip), so its fixture uses LVARCHAR. tests/databases.py gbase8s now passes connection, DDL/DML/query, numeric precision, duplicate columns and first-error stop against the vendor image.
tests/databases.py now says which fixture each of the four vendor-dependent engines needs and why, the GBase 8s reference points at LVARCHAR instead of VARCHAR, the Informix reference explains the user@host authorization the developer image enforces, and the README walks the GBase 8s fixture with its driver variable.
The developer image ignores DB_INFORMIX_PASSWORD and keeps the documented default, so every other identity answered "is not known on the database server". With it, tests/databases.py informix passes connection, DDL/DML/query, numeric precision, duplicate columns and first-error stop on a fresh container.
CI showed the Kylin job failing with "worker emitted an invalid protocol event": the Avatica client prints to standard output while answering a query, and that line lands in the event stream. The worker now keeps its own handle on standard output and sends anything a driver prints there to standard error. Kylin also serves SQL over pre-built cubes, so it takes SELECT and SHOW but no DDL or DML. Its fixture reads the sample project the image loads and checks the cube query plus the first-error stop instead of creating a table, and it uses a safe column alias because Kylin reserves "value" as well.
design.md lists every engine and its backend and now names the engines whose vendor does not allow redistribution together with sqlx driver add; the Skill's CLI reference tells an agent how to relay that command instead of guessing a download URL.
The Kylin job still failed after the worker stopped letting drivers write to standard output: Avatica reaches the real file descriptor, so the line arrives anyway. The CLI now reports such a line on standard error and keeps reading instead of failing the request, which keeps a chatty driver from turning a good result into "worker emitted an invalid protocol event". A test with a worker that prints an unrelated line covers it.
CI got past the protocol problem and reported the next real one: javax/xml/bind/DatatypeConverter, which the JDK dropped in Java 11 and the pinned JRE 17 therefore does not provide. The Kylin component now carries the JAXB API and runtime, their istack and activation dependencies, and an slf4j 1.7 binding, so the driver starts and reaches the network -- verified locally against a closed port, which now fails with a connection error instead of a missing class.
Kylin's sample project did not expose the table the fixture named, so the read-only path now asks the engine with SHOW TABLES and counts the first table it reports. The fake worker in the CLI test also reads its request now: a worker that exits without reading makes the writer see EPIPE, which is what the arm64 runner reported while macOS passed by timing.
The packaged license URL 404s, which the release step would have hit. jaxb-runtime carries the Eclipse Distribution License and its notice inside the jar, which is the source the packagers already prefer for drivers that ship one.
Its 5.0.2 image refuses SHOW TABLES with "This SQL is not supported at the moment", so the read-only fixture no longer depends on a catalog listing or on the sample project's table names. It checks two literal queries and the first-error stop, and references/kylin.md records the version caveat.
The clap list, the help text and the MCP schema are three copies of the same set, and the new engines had to be added to each by hand. A unit test now parses every documented component and checks the help names it, so a missing entry fails the build instead of a user's command.
* A driver set is validated before anything is copied, so a rejected jar no longer leaves a partial install behind, and the set is accepted when one of its jars carries the driver class: a driver that needs dependencies can be provided with them. * A file that is not named as a jar is refused instead of stored and silently ignored, and the driver directory is read case-insensitively, matching how it is written. * Released driver components are platform independent and live under `any`; `sqlx driver list` now finds them, which the installed/source fields were reporting wrong. * A provided jar is loaded before the released one it replaces instead of replacing the whole component, so an overridden Kylin driver keeps the JAXB runtime and logging binding its component supplies. * The packaging guard checks every driver entry point in a component, not the first one it finds, and the messages name the wrapper jar GBase 8s ships and the .jar naming requirement.
Coverage and behaviour: * The development worker directory now loads a driver installed with `sqlx driver add` before the jars beside it, so a development run and a release run share one classpath, and tests/databases.py installs the staged jar through the CLI for the engines the release cannot ship, which exercises that path in CI. * `connectionIsValid` tolerates only a driver that says the call is unsupported; any other failure is a failed test again, because a connection test runs no statement that could report it later. * Hive declares the slf4j binding its component ships, so the development classpath matches the release one. * The line a chatty driver writes to the protocol stream is redacted like every other worker output, so a credential a driver logs cannot reach the terminal. Configuration and documentation: * The setup page and the datasource summary ask for the service name Informix and GBase 8s need, which the UI could not provide before. * README, design.md, the Skill's downloads reference and both plugin descriptions name the new engines and the driver command; the prefetch list matches the CLI again. * A guard asserts every documented component is reachable through `sqlx prefetch all`, and a rejected driver jar is asserted not to be installed. scripts/release-driver-check.sh runs the release path against a published manifest. It cannot run in CI before a release carries the new engine, so it is documented for the release step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Eight engines, in two groups.
Bundled through the JDBC worker (driver downloaded with the release, like Oracle and SQL Server):
Presto, Hive, Apache Kylin, XuguDB.
Connected with a driver the user provides, because the vendor does not allow redistribution:
IBM Db2, IBM Informix, SUNDB, GBase 8s.
sqlx driver add --type db2 --jar ~/Downloads/jcc-12.1.0.0.jar sqlx driver list sqlx driver remove --type db2addcopies the jar into<data-dir>/drivers/<engine>/after checking it really carries the engine'sdriver class, a provided jar wins over a released component, and running a statement without a driver is
refused before anything is downloaded, naming the exact command. README documents where each driver comes
from and which class it must contain.
Per-engine work
Protocol enum and
--typenames, driver component/class table, JDBC URL template, the Informix andGBase 8s parameter block for their server instance, default ports, prefetch choices, release packaging
(
scripts/package-shared.py), test fixtures (tests/compose.yaml,scripts/jdbc-fixture.sh,tests/databases.py), a CI job, README, UI engine lists, and one Skill reference page per engine.Two bugs in the JDBC worker were found by running against real servers and are fixed here:
getMoreResultsended a successful statement with a spurious"Method not supported" error (Hive).
NoClassDefFoundErrorinstead of reporting the SQL error. Atry-with-resources closes its resourcebefore an attached
catchruns, so the failure is now reported from inside the loader scope.Db2 has no FROM-less
SELECT, so the shared one-row probe intests/databases.pyis per engine.Verification
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace— clean.mvn -f java/jdbc/pom.xml verify— the new URL templates have unit tests.npm --prefix ui run buildandpython3 tests/distribution.py(skill index, version consistency, 61manifest entries, release contract) — clean.
python3 scripts/package-shared.py --version 0.1.16— builds all 17 components including the four newdrivers. It now also refuses a driver compiled for a newer Java than the pinned JRE 17, which is how the
Hive driver version was chosen (4.0.1; 4.2.x is Java 21).
python3 tests/databases.py presto hive db2 informix gbase8sagainst live fixtures — connection,DDL/DML/query, numeric precision, duplicate column labels and first-error stop all pass.
jdbc-enginesCI job runs Presto, Hive, Kylin and Db2 on an amd64 runner and passes, which is howKylin was verified: its container does not come up under arm64 emulation here.
Running against those servers changed the code, which is why they are worth listing: Hive does not
implement
getMoreResults, GBase 8s does not implementisValidorgetMoreResultsand reports both asa plain
SQLException, GBase 8s fails inside its own URL parser without a client locale, and Informix andGBase 8s have neither
DROP TABLE IF EXISTSnor a FROM-lessSELECT. GBase 8s also reads aVARCHARcolumn back empty with driver 3.70.1.61 while
CHARandLVARCHARround-trip, so its fixture and itsreference page say to store varying text in
LVARCHAR. Informix's developer image ignoresDB_INFORMIX_PASSWORDand keeps its documented default, which is what its fixture now uses. Kylin'sAvatica client prints to standard output while answering a query, which corrupted the event stream until
the worker started keeping its own handle on standard output and the CLI started reporting such a line on
standard error instead of failing the request. The same driver then asked for JAXB, which the JDK dropped
in Java 11, so its component carries the JAXB runtime and an slf4j 1.7 binding.
Not verified on this machine
Six of the eight engines are verified end to end: Presto, Hive, Db2 and Informix against local
containers, GBase 8s against the vendor image with its own driver, and Kylin in CI.
XuguDB and SUNDB cannot be verified with public assets at all — the XuguDB trial image does not publish its
SYSDBA password and its driver has no trust mode, and the SUNDB image license expired in 2022. Both ship
with driver mapping, fixture, probe, Skill page and the environment variable
tests/databases.pydocuments, so a licensed instance or a known password completes the check without code changes.
The matching Chat2DB engine mapping for these eight types lands in the Chat2DB repository.