data-structures: a publish keeps a column its table no longer declares, drops it only when listed as dropped, and renames it in place when declared renamedFrom (#7635)

[delchev]

Cause

TableAlterProcessor, the alter path of every .table and .schema (so of every intent-generated table), reconciled the live table in two passes. First it ADDed the columns the model declares, then it DROPped every live column the model no longer names. TablesSynchronizer / SchemasSynchronizer are multitenant, so the DROP ran in every tenant schema, and the only log line was the INFO Processing Alter Table. A renamed intent field emits a new column name, so the publish added the new column and dropped the old one with its data, on the first boot of the new image. An image-only rollback cannot undo that.

Change: expand/contract for undeclared columns

• Kept by default. An undeclared column stays with its data. It is logged at WARN with the table, the column and the owning artefact, and counted as the orphanColumns detail of the artefacts health component (PlatformReadiness.recordOrphanColumns, keyed per catalog.schema.table so one tenant's copy does not overwrite another's). A kept NOT NULL column is relaxed to nullable, because the app no longer writes it and every insert would otherwise be refused (new ISqlDialect.dropNotNull). MySQL, MariaDB, MSSQL and HANA return null there, because they need the whole column type restated; they get a WARN saying inserts that omit the column will be refused.
• Dropped only when declared. A table's dropped: [...] removes those columns. DIRIGIBLE_DATABASE_DROP_UNDECLARED_COLUMNS=true is an operator opt-in for dev instances.
• Rename as a first-class change. A column's renamedFrom renames the live one in place before the ADD pass (new ISqlDialect.renameColumn: ANSI by default, HANA's RENAME COLUMN t.c TO n). On MSSQL and MongoDB it returns null, and the ADD pass then adds the column nullable, copies the values and keeps the old one. A rename runs only while the old name is live and the new one is not, so it is harmless to leave in the definition.
• The INCOMPATIBLE_CHANGE_OF_TABLE refusals are unchanged.

Intent DSL. A field's renamedFrom: <old name> and an entity's dropped: [names] become the .model's dataRenamedFrom / dataDropped. These are the former column names, derived like dataName, and comma-separated so they survive the scalar-only .edm. template-application-schema emits them as the column's renamedFrom and the table's dropped. The parser refuses:

• either declaration naming a column the entity still declares (compared by column, so a relation counts);
• a rename from the field's own name or from a dropped name;
• two fields renamed from one name.

The assistant guide, intent-dsl-features.md and synchronizer-model.md document it.

Verification

Run:

• Unit: TableAlterProcessorTest (keep + count + relaxed NOT NULL with a later insert, declared drop, operator opt-in, in-place rename with a no-op second run), ColumnChangeStatementTest (default / MySQL / HANA), PlatformReadinessTest, SchemaEvolutionValidationTest, EdmIntentGeneratorTest. Every changed module's unit suite is green.
• TableUndeclaredColumnIT (new, HTTP-level, untagged so it is in the PR smoke set) on H2 and on a local PostgreSQL 16. pg_stat_database confirmed PostgreSQL was written to. The run:
  • publish A, B NOT NULL with a row, then republish with A only: B and its value survive, the WARN is logged, the orphan is counted, and an insert without B passes;
  • republish with dropped: [B]: B is gone;
  • the schema the real template renders from a .model with dataRenamedFrom / dataDropped, published over the old table: the values moved to the new name, and the old and the dropped columns are gone.
• SchemaRepublishTypeToleranceIT (H2 + PostgreSQL), SchemaTemplateUniqueConstraintIT, SchemaTwoUniqueConstraintsIT and IntentEngineIT (81 tests) on H2.
• formatter:validate with the cache wiped, and the release-profile javadoc build of the touched modules.

Not run:

• The ADD + copy fallback and the MySQL/MSSQL/HANA paths against a real database (only the statement shapes are unit-tested); none of those databases is a CI leg.
• A second tenant: the IT runs in the default tenant. The per-tenant isolation of the orphan count is unit-tested only.
• The full nightly suite.

Not in this PR: the intentfile.org spec/site and dirigible.io /help/intent/ pages for renamedFrom: / dropped: follow as separate PRs.

Fixes #7635

🤖 Generated with Claude Code