data-structures: a publish keeps a column its table no longer declares, drops it only when listed as dropped, and renames it in place when declared renamedFrom (#7635)

.claude/docs/intent-dsl-features.md

@@ -18,6 +18,8 @@ Loaded on demand, not on every session: read this before changing a DSL construc
 
 **The declarative state machine (`lifecycle:`, [#6714](https://github.com/eclipse-dirigible/dirigible/issues/6714)):** an entity may declare the WHOLE set of legal status edges over its `function: EntityStatus` nomenclature — `lifecycle: { edges: [{ from: DRAFT, to: [ISSUED, CANCELLED] }, ...] }`, either side a seeded name or an id — and every status write is validated against it. Until then the status machinery was point constructs (`init:`, a `transitions:` button's own guard, a workflow `setRelationField`, a check's rejection) with nothing stating which moves were legal at all, so any writer that was not a transition button could move a document from any status to any other. Enforcement is in the generated **repository** — the one choke point every writer passes through (`update`, `updateWithoutEvent`, and `updateProperties`, which the transition controller's `updateProperty` and the workflow setters route through) — rejecting an unmodeled move with 400 and a message naming both statuses; with `init:` declared, a record cannot be CREATED mid-lifecycle either. At parse time the graph is what the other status sites are held to: a `transitions:` entry's `from`→`setStatus` pair must be a declared edge (a button is presentation over the graph), and a status written by a workflow step or forced by a check must be one some edge reaches — so a reject path transiting through an approved status fails when the intent is read. There is no `on:` key (the graph is always over the EntityStatus relation, and YAML reads a bare `on` as `true`, so it is refused rather than silently dropped), and a cross-model nomenclature is declared where it is seeded. Details in the engine-intent guide's state-machine bullet.
 
+**Expand/contract on a live table (`renamedFrom:` / `dropped:`, [#7635](https://github.com/eclipse-dirigible/dirigible/issues/7635)):** a publish no longer drops a column its definition stops declaring - `TableAlterProcessor` keeps it with its data in every tenant, WARNs naming the owning artefact, relaxes a kept `NOT NULL` (dialect `dropNotNull`, `null` on MySQL/MariaDB/MSSQL/HANA, where it would need the whole type: then only the WARN), and counts it as the `orphanColumns` detail of the `artefacts` health component (`PlatformReadiness.recordOrphanColumns`, per catalog.schema.table so tenants do not overwrite each other). A field's `renamedFrom: <old name>` becomes the `.model` property's `dataRenamedFrom` (the old COLUMN, derived like `dataName`) and the schema column's `renamedFrom`, and the alter renames the live column in place before the ADD pass (dialect `renameColumn`; on MSSQL/MongoDB it returns `null` and the ADD pass adds the column nullable, copies the values and keeps the old one); an entity's `dropped: [names]` becomes `dataDropped` (comma-separated - it must survive the scalar-only `.edm`) and the table's `dropped`, the only way besides the dev opt-in `DIRIGIBLE_DATABASE_DROP_UNDECLARED_COLUMNS=true` that a column leaves. Both act only while there is something to do, so they are harmless to leave in the intent. The parser refuses either one naming a column the entity still declares (compared by column, so a relation counts), a rename from its own name or from a dropped name, and two renames from one name. Unique constraints are still reconciled by drop - they hold no data.
+
 **A field's label, and a label the TENANT'S COUNTRY resolves (`label:` / `countryLabels:`, [#6424](https://github.com/eclipse-dirigible/dirigible/issues/6424)):** a field may now declare its display `label:` - emitted as the property's own `widgetLabel`, which every generated surface renders and the en-US catalog is seeded from, so an acronym or a unit (`nationalId` as "National ID", not the humanized "National Id") is expressed in the intent instead of hand-edited into a catalog the next Generate overwrites. Alongside it, `countryLabels: { BG: ЕГН, DE: Steuer-ID }` declares variants resolved from the **tenant's country** (`DIRIGIBLE_APPLICATION_COUNTRY`, ISO 3166-1 alpha-2, tenant-overridable in the application shell's Tenant Configuration) rather than from the UI language - which term a national identifier goes by is a property of the company, so keying it off the language catalogs is wrong in both directions at once (the Bulgarian-reading user of a German company gets the local term; the English-reading accountant of a Bulgarian one gets the generic one). It also cannot live in a catalog mechanically: the shared `i18n.js` does not load catalogs at all in the default language. So the variants travel as a language-independent overlay - a structured `widgetCountryLabels` on the property (Map, hence `.model`-only), flattened at UI generation into the `countryLabels` object `config.js` carries, keyed by the very translation key the views bind, which `T()` consults ahead of both i18next and the baked fallback in every language. An app declaring no variant issues no extra request and generates byte-identically. A key that is not a country is refused at parse time (it could never match a tenant); report column labels are deliberately out of scope, a column alias being its SQL alias too.
 
 **Naming a rendered document (`fileName:`, [#6899](https://github.com/eclipse-dirigible/dirigible/issues/6899)):** the two server-side PDF renders — the snapshot copy a document mints on issue and the `attach: print` PDF a `notify` block sends — accept a **`fileName:`** pattern on the `function: Snapshot` child / inside the notify block: literals plus `{token}` interpolations, no expression language. `{field}` and one-hop `{relation.field}` use the same path vocabulary and **authored names** a notify subject does, `{field:pattern}` formats a `date`/`timestamp` through a `DateTimeFormatter` pattern, `{A|B}` renders the first non-blank operand, `{Version}` (snapshot only) places the copy's version and a pattern without it gets `_v<n>` appended. Interpolated **values** are sanitized at run time by `sdk.print.FileNames` (trim, whitespace → one `_`, path/control characters stripped, non-ASCII deliberately kept); the literal separators are the author's. A pattern's relation loads are **merged into the notify block's own**, deduped by local, so a name may read a relation the message never mentions — but `attach: recordPrint` renders the anchor once, before the per-row loop, so only fields of the anchor are readable there. Absent a pattern, the snapshot default **changes** from the old primary-key form (`Order 42 v1.pdf`) to the same number-or-id expression the mail already used, plus the version — the deliberate fix for the two names disagreeing. An unresolvable path, an unbalanced brace, a format on a non-date field or a pattern that interpolates nothing are validation errors. Details in the engine-intent guide's fileName bullet.

.claude/docs/synchronizer-model.md

@@ -19,3 +19,5 @@ This does **not** retire `RegistryMutationTracker`: a pass is now scheduled whil
 **A failure of a collaborator records `FAILED` and is retried; `FATAL` is not a synchronizer's answer to it** (#7248). A `.listener` whose subscription the embedded broker refused while it was still taking its store lease, a `.job` the scheduler could not schedule yet, a `.camel` route missing a bean, a `.schema` naming a `.datasource` published on a later pass - each of these is transient, and each of the four synchronizers used to promote the second failure to `FATAL`, after which `SynchronizationProcessor.parseDefinitions` strips the artefact from every later pass until the file's bytes change (a topic subscription lost to a boot race stayed lost, with every message discarded). The shape now, in all four: a failed start registers `FAILED` **with its cause** (never `CREATED` while nothing runs) and returns `false`, which hands the artefact to the in-pass cross-retry loop (`DIRIGIBLE_SYNCHRONIZER_CROSS_RETRY_COUNT` x `_INTERVAL_MILLIS`, so a permanently failing artefact costs that loop every pass - the same cost a `FAILED` view carries since #6942); the `START` phase retries it and heals it to `CREATED` on success. **A pass runs its phases only when it carries a `NEW`/`MODIFIED` artefact** (`isSynchronizationNeeded` answers "the registry changed", and the test framework waits on that answer for a quiet period - do not widen it), so the processor additionally gives every `FAILED` artefact ONE `START` attempt per pass on an idle instance every `DIRIGIBLE_SYNCHRONIZER_FAILED_RETRY_INTERVAL_SECONDS` (30) via a private gate (`isFailedRetryDue` / `retryFailed`) - no cross-retry loop, and a repeat of the same error logs at DEBUG rather than stack-tracing at ERROR on every attempt. Two multitenant details: `MultitenantBaseSynchronizer` completes one shared artefact once per tenant and resets only the lifecycle between tenants, so a retry gate must include `FAILED` and not rely on `running` alone (the first tenant to subscribe flips it and would skip the rest), and the managers' idempotence per tenant (`ListenersManager.LISTENERS.containsKey`, `JobsManager.scheduleJob`'s `checkExists`) is what makes the repeated attempt safe. `ListenersManager.startListener` throws for a missing handler rather than returning normally - a normal return read as CREATED and running with nothing subscribed. Artefacts already persisted as `FATAL` by an earlier version stay stripped until republished; `case BROKEN:` in `parseDefinitions` is the same philosophy one level down (a definition that failed to parse is re-parsed every pass).
 
 **A `.bpmn` DELETE retires its deployments, it never cascades (#7597).** `BpmnSynchronizer.removeFromProcessEngine` used to call Flowable's `deleteDeployment(id, true)`, which destroys every running instance of the definition together with its user tasks AND their history - so a `.bpmn` that was absent for one pass (the cleanup pass reaps any artefact whose source file is missing from the registry, which is what a fresh container sees before its module content has landed) took a month of approvals with it and left nothing to prove it. `BpmProviderFlowable.retireDeployment` now deletes a deployment only when none of its definitions has a running instance, non-cascading, so the finished instances stay in the history; one with running instances is kept with its definitions **suspended** (a WARN names the key, the deployment and the instance ids), so no new instance starts from it while the running ones complete. The artefact is still registered DELETED; should it come back, `deployProcess` is simply the next version of the same key (new instances start from it) and sweeps the retired version once its last instance has ended - as does the next DELETE. The explicit SDK `Deployer.undeployProcess` refuses with an `IllegalStateException` naming the running instances instead (its javadoc always promised not to terminate them). The history level is now set explicitly (`DIRIGIBLE_FLOWABLE_HISTORY_LEVEL`, default `audit`, Flowable's own default; `none` is accepted as an opt-out, logged at WARN, and shown on the monitoring shell's Processes page via `GET /services/bpm/bpm-processes/engine`), every instance deletion logs its id, definition and reason before the engine acts, and `BpmnDeploymentRetireIT` + `BpmProviderFlowableRetireDeploymentTest` pin the lifecycle. One Flowable detail the IT surfaced: `historic-instances?definitionKey=` is answered through a join on `ACT_RE_PROCDEF`, so the history of a definition whose deployments are all gone is listed unfiltered and by `businessKey`, never by `definitionKey`.
+
+**A table publish never drops a column on its own (#7635).** `TableAlterProcessor` (the alter path of both `.table` and `.schema`, so of every intent-generated table) keeps a live column its definition no longer declares, with its data, in every tenant - WARN, a relaxed `NOT NULL`, and the `orphanColumns` detail of the `artefacts` health component. A column leaves only when the table lists it under `dropped` (intent: `dropped:`), or on a dev instance with `DIRIGIBLE_DATABASE_DROP_UNDECLARED_COLUMNS=true`; a column's `renamedFrom` (intent: `renamedFrom:`) renames the live one in place. Do not reintroduce a silent DROP: a rename regenerated as add-new/drop-old lost the column's data on the first boot of a rolled image, and the rollback is image-only.

components/core/core-base/src/main/java/org/eclipse/dirigible/components/base/readiness/PlatformReadiness.java

@@ -11,7 +11,10 @@
 
 import java.time.Instant;
 import java.util.List;
+import java.util.Map;
 import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArrayList;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
@@ -58,6 +61,7 @@ public enum State {
     private volatile Instant since = Instant.now();
     private volatile ArtefactCensus artefacts = ArtefactCensus.NONE;
     private volatile CompiledModulesCensus compiledModules = null;
+    private final Map<String, Set<String>> orphanColumns = new ConcurrentHashMap<>();
 
     private PlatformReadiness() {}
 
@@ -127,6 +131,34 @@ public ArtefactCensus getArtefacts() {
         return artefacts;
     }
 
+    /**
+     * Records the columns a table keeps in the database although its published definition no longer
+     * declares them (#7635) - kept with their data instead of dropped. Each reconciliation of the table
+     * replaces its previous record, so a column that is declared again, or dropped, stops counting.
+     *
+     * @param table the table, qualified by its catalog and schema - one tenant's copy is not another's
+     * @param columns the undeclared columns, empty when the table has none
+     */
+    public void recordOrphanColumns(String table, Set<String> columns) {
+        if (columns.isEmpty()) {
+            orphanColumns.remove(table);
+        } else {
+            orphanColumns.put(table, Set.copyOf(columns));
+        }
+    }
+
+    /**
+     * The undeclared columns kept across every table, as the last reconciliation of each left them.
+     *
+     * @return the orphan column count
+     */
+    public int getOrphanColumns() {
+        return orphanColumns.values()
+                            .stream()
+                            .mapToInt(Set::size)
+                            .sum();
+    }
+
     /**
      * Records what the last AOT compiled-module discovery found (#7533). It runs on the application
      * ready event, independently of the synchronization passes, so the listeners are notified with the
@@ -185,6 +217,7 @@ public void reset() {
         since = Instant.now();
         artefacts = ArtefactCensus.NONE;
         compiledModules = null;
+        orphanColumns.clear();
         listeners.clear();
     }
 

components/core/core-base/src/test/java/org/eclipse/dirigible/components/base/readiness/PlatformReadinessTest.java

@@ -16,6 +16,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 
 import org.eclipse.dirigible.components.base.readiness.PlatformReadiness.State;
 import org.junit.jupiter.api.BeforeEach;
@@ -90,4 +91,17 @@ void theCompiledModulesCensusNotifiesTheListeners() {
 
         assertEquals(List.of(State.INITIALIZING), notified, "a consumer waiting for a clean boot must re-evaluate");
     }
+
+    @Test
+    void orphanColumnsAreCountedPerTableAndAReconciliationReplacesItsTablesRecord() {
+        readiness.recordOrphanColumns("DB.TENANT_A.INVOICE", Set.of("OLD_NOTE", "LEGACY_CODE"));
+        readiness.recordOrphanColumns("DB.TENANT_B.INVOICE", Set.of("OLD_NOTE"));
+        assertEquals(3, readiness.getOrphanColumns(), "one tenant's copy of a table does not overwrite another's");
+
+        readiness.recordOrphanColumns("DB.TENANT_A.INVOICE", Set.of());
+        assertEquals(1, readiness.getOrphanColumns(), "a table reconciled without orphans stops counting");
+
+        readiness.reset();
+        assertEquals(0, readiness.getOrphanColumns());
+    }
 }

components/core/core-healthcheck/src/main/java/org/eclipse/dirigible/components/base/healthcheck/indicator/ArtefactsHealthIndicator.java

@@ -18,9 +18,11 @@
 /**
  * The {@code artefacts} health component (#7533): the lifecycle census of every registered artefact
  * as the last synchronization pass left it - how many there are, how many are failed, and of which
- * type. A failed artefact is a quality signal, not an outage, so it never turns the component DOWN;
- * a gate reads {@code failed}, and {@code DIRIGIBLE_READINESS_REQUIRE_CLEAN_BOOT} is the opt-in
- * that makes it withhold readiness. UNKNOWN until the first pass has depleted.
+ * type, plus how many database columns are kept although their table's definition no longer
+ * declares them ({@code orphanColumns}, #7635). A failed artefact is a quality signal, not an
+ * outage, so it never turns the component DOWN; a gate reads {@code failed}, and
+ * {@code DIRIGIBLE_READINESS_REQUIRE_CLEAN_BOOT} is the opt-in that makes it withhold readiness.
+ * UNKNOWN until the first pass has depleted.
  */
 @Component
 class ArtefactsHealthIndicator implements HealthIndicator {
@@ -35,6 +37,7 @@ public Health health() {
                       .withDetail("total", census.total())
                       .withDetail("failed", census.failed())
                       .withDetail("failedByType", census.failedByType())
+                      .withDetail("orphanColumns", readiness.getOrphanColumns())
                       .build();
     }
 }

components/data/data-structures/src/main/java/org/eclipse/dirigible/components/data/structures/domain/Table.java

@@ -10,12 +10,14 @@
 package org.eclipse.dirigible.components.data.structures.domain;
 
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.List;
 import java.util.Set;
 
 import jakarta.annotation.Nullable;
 import jakarta.persistence.CascadeType;
 import jakarta.persistence.Column;
+import jakarta.persistence.Convert;
 import jakarta.persistence.Entity;
 import jakarta.persistence.FetchType;
 import jakarta.persistence.GeneratedValue;
@@ -27,6 +29,7 @@
 import jakarta.persistence.OneToOne;
 
 import org.eclipse.dirigible.components.base.artefact.Artefact;
+import org.eclipse.dirigible.components.base.converters.ArrayOfStringsToCsvConverter;
 import org.hibernate.annotations.OnDelete;
 import org.hibernate.annotations.OnDeleteAction;
 
@@ -76,6 +79,16 @@ public class Table extends Artefact {
     @Expose
     private TableConstraints constraints;
 
+    /**
+     * The columns this table no longer has (#7635): a live column the definition does not declare is
+     * kept, with its data, unless it is named here - the explicit contract step of a schema change.
+     */
+    @Column(name = "TABLE_DROPPED_COLUMNS", columnDefinition = "VARCHAR", nullable = true, length = 2000)
+    @Nullable
+    @Convert(converter = ArrayOfStringsToCsvConverter.class)
+    @Expose
+    private String[] dropped;
+
     /** The schema reference. */
     @ManyToOne(fetch = FetchType.EAGER, optional = true)
     @JoinColumn(name = "SCHEMA_ID", nullable = true)
@@ -283,12 +296,30 @@ public void setSchemaReference(Schema schemaReference) {
      *
      * @return the string
      */
+    /**
+     * Gets the columns this table no longer has.
+     *
+     * @return the dropped column names, or null when none are declared
+     */
+    public String[] getDropped() {
+        return dropped;
+    }
+
+    /**
+     * Sets the columns this table no longer has.
+     *
+     * @param dropped the dropped column names
+     */
+    public void setDropped(String[] dropped) {
+        this.dropped = dropped;
+    }
+
     @Override
     public String toString() {
         return "Table [id=" + id + ", schemaName=" + schema + ", columns=" + columns + ", indexes=" + indexes + ", constraints="
-                + constraints + ", location=" + location + ", name=" + name + ", type=" + type + ", description=" + description + ", key="
-                + key + ", dependencies=" + dependencies + ", createdBy=" + createdBy + ", createdAt=" + createdAt + ", updatedBy="
-                + updatedBy + ", updatedAt=" + updatedAt + "]";
+                + constraints + ", dropped=" + Arrays.toString(dropped) + ", location=" + location + ", name=" + name + ", type=" + type
+                + ", description=" + description + ", key=" + key + ", dependencies=" + dependencies + ", createdBy=" + createdBy
+                + ", createdAt=" + createdAt + ", updatedBy=" + updatedBy + ", updatedAt=" + updatedAt + "]";
     }
 
     /**