ATLAS-5338: Add Trino extractor integration and unit test suite

[ramackri]

Summary

Implements the Trino extractor test suite that was previously only sketched as comments in TrinoExtractorIT. Tests exercise the standalone extractor tarball (run-trino-extractor.sh) against a live Atlas + Trino stack, matching production layout rather than the Maven test classpath.

JIRA: https://issues.apache.org/jira/browse/ATLAS-5338

Changes

TrinoExtractorIT

Full live integration test suite:

• Invalid CLI arguments and cron expression handling
• Catalog, schema, table, and column metadata import
• Unregistered catalog via command-line -c
• Stale entity cleanup after delete/rename (opt-in via TRINO_IT_DELETE_SYNC=1)
• Hive hook linkage to trino_column (opt-in via TRINO_IT_HOOK_ENABLED=1)
• PII tag propagation via hook (opt-in via TRINO_IT_TAG_PROPAGATION=1)
• Tarball Jersey classpath verification

TrinoExtractorITSupport

Shared test infrastructure:

• Tarball workdir preparation and cleanup
• Atlas REST entity existence/absence assertions
• Stale entity seeding for delete-sync scenarios
• Property overrides and extractor script execution helpers

TrinoExtractorTest

Unit tests (no live stack required):

• Invalid/valid Quartz cron expression validation
• @DisallowConcurrentExecution on MetadataJob

pom.xml

• New Maven profile trino-extractor-it: builds extractor tarball in pre-integration-test, runs Failsafe *IT tests
• Test dependencies: testng, commons-io

src/test/assemblies/trino-extractor-tarball.xml

Assembly descriptor for IT tarball packaging (same lib/ layout as distro).

README.md

Documents how to run unit and integration tests, required environment variables, optional flags, coverage map, and components required to run each test tier.

Components required to run tests

What must be up depends on which tests you run. Full detail is in addons/trino-extractor/README.md.

Unit tests only

Required	Not required
JDK + Maven	Atlas, Trino, Kafka, Hive, Ranger

mvn -pl addons/trino-extractor test -Dtest=TrinoExtractorTest

Default integration tests (TrinoExtractorIT)

Set ATLAS_REST_URL and run:

mvn -pl addons/trino-extractor -Ptrino-extractor-it verify

Must be running:

Component	Role	Default
Atlas server	REST API — create/update/assert trino_* entities	http://localhost:21000
Atlas backend store	Postgres or HBase (started with Atlas)	—
Trino coordinator	JDBC metadata source for the extractor	jdbc:trino://localhost:8080/
Hive metastore	Backing store for the Trino hive catalog	e.g. ranger-hive in docker lab
Trino hive catalog	Catalog the extractor queries	hive
Schema / table / column	Objects imported into Atlas	hr.trino_pii_lab.ssn

Trino Extractor IT  --REST-->  Atlas (:21000)
        |
        +--JDBC-->  Trino (:8080)  -->  hive catalog  -->  Hive metastore

Minimum environment variables:

export ATLAS_REST_URL=http://localhost:21000
export TRINO_JDBC_URL=jdbc:trino://localhost:8080/
export ATLAS_USERNAME=admin
export ATLAS_PASSWORD=atlasR0cks!

# Optional — defaults match ranger-docker lab
export TRINO_EXTRACTOR_CATALOG=hive
export TRINO_EXTRACTOR_SCHEMA=hr
export TRINO_EXTRACTOR_TABLE=trino_pii_lab
export TRINO_EXTRACTOR_COLUMN=ssn
export ATLAS_TRINO_NAMESPACE=dev

Not required for default ITs: Kafka, Hive hook, Ranger, TagSync, or a pre-built distro tarball (Maven profile builds the IT tarball).

If ATLAS_REST_URL is unset or Atlas is unreachable, TrinoExtractorIT skips (suite passes without live tests).

Optional integration tests (extra flags)

Flag	Extra components	What it verifies
TRINO_IT_HOOK_ENABLED=1	HiveServer2 with Atlas Hive hook; hive_column in Atlas (default namespace cm)	trino_column links to hive_column
TRINO_IT_TAG_PROPAGATION=1	Hook linkage + Atlas classification API	PII tag propagates to trino_column
TRINO_IT_DELETE_SYNC=1	Atlas only (stale entities seeded via REST)	Stale trino_table / trino_schema / trino_catalog removal

TagSync and Ranger are not required even for tag propagation ITs (Atlas REST only).

Test tier matrix

Tier	Atlas	Trino	Hive MS + table	Hive hook	Kafka	Ranger	TagSync
Unit (TrinoExtractorTest)	—	—	—	—	—	—	—
Default IT	Yes	Yes	Yes (hr.trino_pii_lab)	—	—	—	—
Hook IT	Yes	Yes	Yes	Yes	—*	—	—
Tag propagation IT	Yes	Yes	Yes	Yes	—*	—	—
Delete sync IT	Yes	Yes**	—	—	—	—	—

* Kafka only if your Hive hook setup uses it; not required by the extractor itself.
** Trino still needed for extractor JDBC; stale entities are seeded via Atlas REST.

Recommended docker lab (default ITs)

Atlas coexist + ranger-docker stack:

Service	Port	Purpose
atlas	21000	Atlas REST + graph store
Atlas Postgres (or HBase)	—	Atlas persistence
ranger-trino	8080	Trino JDBC
ranger-hive	—	Hive metastore for hive catalog

Create the test table if missing:

CREATE TABLE hr.trino_pii_lab (id INT, ssn STRING, address STRING);

Pre-flight checks:

curl -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/trino_column
docker exec ranger-trino trino --user admin --execute "SHOW SCHEMAS FROM hive"
docker exec ranger-trino trino --user admin --execute "DESCRIBE hive.hr.trino_pii_lab"

How to test

Unit tests (no Atlas/Trino required):

mvn -pl addons/trino-extractor test -Dtest=TrinoExtractorTest

Live integration tests:

export ATLAS_REST_URL=http://localhost:21000
export TRINO_JDBC_URL=jdbc:trino://localhost:8080/
export ATLAS_USERNAME=admin
export ATLAS_PASSWORD=atlasR0cks!
export TRINO_EXTRACTOR_SCHEMA=hr
export TRINO_EXTRACTOR_TABLE=trino_pii_lab
export TRINO_EXTRACTOR_COLUMN=ssn

mvn -pl addons/trino-extractor -Ptrino-extractor-it verify

Optional flags:

Variable	Purpose
TRINO_IT_HOOK_ENABLED=1	Verify trino_column links to hive_column
TRINO_IT_TAG_PROPAGATION=1	Verify PII tag propagation
TRINO_IT_DELETE_SYNC=1	Verify stale catalog/schema/table removal

Test plan

• mvn -pl addons/trino-extractor test -Dtest=TrinoExtractorTest passes
• mvn -pl addons/trino-extractor -Ptrino-extractor-it verify against Atlas + Trino lab (manual; requires ATLAS_REST_URL)
• ITs skip gracefully when live stack is not configured

Sync with master

• Merged origin/master (includes ATLAS-5336 Kafka 3.9.1 / KRaft and ATLAS-5337 Trino Jersey client fix).
• Resolved addons/trino-extractor/pom.xml conflict (kept opt-in trino-extractor-it profile).

JDK 8 CI fix

TrinoExtractorITSupport used List.of() (Java 9+), which broke testCompile on the CI JDK 8 matrix job. Replaced with Arrays.asList() / Arrays.toString().

HBase Docker CI hardening (compose smoke test)

CI Docker image builds were succeeding; failures were in Bring up containers when atlas-hbase was marked unhealthy (~27s on slow runners). HBase is only healthy when both endpoints respond:

• Master: http://localhost:16010/master-status
• RegionServer: http://localhost:16030/rs-status

RegionServer often starts after Master; the old start_period: 40s, interval: 30s healthcheck and tail --pid=HMaster entrypoint were flaky.

Fix (this PR):

File	Change
scripts/atlas-hbase-healthcheck.sh	Shared Master + RS probe (wget/curl)
scripts/atlas-hbase.sh	Wait up to 180s for both endpoints; keep alive while Master or RS JVM runs
docker-compose.atlas-hbase.yml	start_period: 120s, interval: 10s, use healthcheck script
Dockerfile.atlas-base	Install wget + curl
Dockerfile.atlas-hbase	Copy + chmod healthcheck script
.github/workflows/ci.yml	--wait-timeout 300; dump service logs when compose up --wait fails

Local verify (atlas-docker lab):

cd dev-support/atlas-docker
docker compose -f docker-compose.atlas-base.yml build
docker compose -f docker-compose.atlas.yml \
  -f docker-compose.atlas-hadoop.yml \
  -f docker-compose.atlas-hive.yml up -d --wait --wait-timeout 300
curl -sf http://localhost:16010/master-status && curl -sf http://localhost:16030/rs-status
docker inspect -f '{{.State.Health.Status}}' atlas-hbase   # expect: healthy

Note: ports 16010/16030 are exposed only when the atlas-docker atlas-hbase container is running (ATLAS_BACKEND=hbase).
A Ranger-only or Postgres-backed Atlas stack does not expose them.

ImportTaskListenerImplTest stabilization (JDK 8 CI flake)

CI run 28991499552 failed on JDK 8 with a deterministic-looking but intermittent unit-test failure (not related to Trino or HBase Docker changes):

• Test: ImportTaskListenerImplTest.testStartImportConsumer_Successful
• Error: ClassCastException: ImportStatus cannot be cast to String at getImportId()

Root cause: the test injected a real ExecutorService while using a shared Mockito mock for AtlasAsyncImportRequest. The async worker thread accessed mock stubs concurrently with the main thread during onReceiveImportRequest(), corrupting stub return types.

Fix (030af4da1):

Change	Why
createImportRequestMock() helper	Per-test isolated mock instead of shared instance
synchronousExecutor() helper	Runs startImportConsumer on the calling thread; no concurrent mock access
Fresh mock in @BeforeMethod	Avoids stale stubs after Mockito.reset()
Failure test: poll returns id once then null	Prevents re-processing loop when using sync executor

Local verify:

mvn -pl webapp -DskipDocs=true -DskipEnunciate=true -DskipITs=true -Drat.skip=true \
  -Dtest=ImportTaskListenerImplTest test