Fix ENTSO-E price curves and gaps, and handle unavailable tariffs in EMS

ems/docs/optimisationMethods/EmsOptimisation.md

@@ -1 +1,7 @@
 Work in progress
+
+## Unavailable tariff forecasts
+
+Both `EmsOptimisation` and `EmsOptimisationBeta` treat null tariff values as unavailable. A moving-average window is eligible for tariff-based charging or discharging only when all its prices are available (the default window contains eight datapoints). Windows containing nulls are excluded; available windows before or after them can still be selected at their original timestamps.
+
+Import and export forecasts are evaluated independently. If no complete import window exists for a day, no tariff-based charging zone is selected for it; the same applies to export windows and discharging. If neither has a complete window, tariff-based zone selection produces no zones for that day. Other optimisation logic continues. Missing prices are not replaced with zero, interpolated, or carried forward.

ems/src/main/java/org/openremote/extension/ems/manager/optimisationMethods/EmsOptimisation.java

@@ -2154,6 +2154,7 @@ private Map<Long, Integer> calculateTariffChargeAndDischargeZones(
 
     Map<LocalDate, IndexedDatapoint> tariffImportDailyMinimumMap =
         IntStream.range(0, tariffImportMovingAverage.size())
+            .filter(i -> tariffImportMovingAverage.get(i).getValue() != null)
             .mapToObj(
                 i -> {
                   ValueDatapoint<?> dp = tariffImportMovingAverage.get(i);
@@ -2167,6 +2168,7 @@ private Map<Long, Integer> calculateTariffChargeAndDischargeZones(
 
     Map<LocalDate, IndexedDatapoint> tariffExportDailyMinimumMap =
         IntStream.range(0, tariffExportMovingAverage.size())
+            .filter(i -> tariffExportMovingAverage.get(i).getValue() != null)
             .mapToObj(
                 i -> {
                   ValueDatapoint<?> dp = tariffExportMovingAverage.get(i);
@@ -2397,24 +2399,30 @@ public List<ValueDatapoint<?>> movingAverage(List<ValueDatapoint<?>> dataPoints,
     }
 
     double sum = 0.0;
+    int unavailableCount = 0;
 
     for (int i = 0; i < dataPoints.size(); i++) {
       Double value = (Double) dataPoints.get(i).getValue();
-      sum += value;
+      if (value == null) {
+        unavailableCount++;
+      } else {
+        sum += value;
+      }
 
       if (i >= window) {
         Double valueOld = (Double) dataPoints.get(i - window).getValue();
-        sum -= valueOld;
+        if (valueOld == null) {
+          unavailableCount--;
+        } else {
+          sum -= valueOld;
+        }
       }
 
       if (i >= window - 1) {
-        double avg = sum / window;
-
-        ValueDatapoint<?> original = dataPoints.get(i);
-
-        ValueDatapoint<Double> averaged =
-            new ValueDatapoint<>(original.getTimestamp(), round(avg, 7));
-        result.add(averaged);
+        // Keep unavailable windows in the result so indices still map to the input tariffs.
+        // A partial window must not be used to select a charge or discharge zone.
+        Double avg = unavailableCount == 0 ? round(sum / window, 7) : null;
+        result.add(new ValueDatapoint<>(dataPoints.get(i).getTimestamp(), avg));
       }
     }
 

ems/src/main/java/org/openremote/extension/ems/manager/optimisationMethods/EmsOptimisationBeta.java

@@ -2204,6 +2204,7 @@ private Map<Long, Integer> calculateTariffChargeAndDischargeZones(
 
     Map<LocalDate, IndexedDatapoint> tariffImportDailyMinimumMap =
         IntStream.range(0, tariffImportMovingAverage.size())
+            .filter(i -> tariffImportMovingAverage.get(i).getValue() != null)
             .mapToObj(
                 i -> {
                   ValueDatapoint<?> dp = tariffImportMovingAverage.get(i);
@@ -2217,6 +2218,7 @@ private Map<Long, Integer> calculateTariffChargeAndDischargeZones(
 
     Map<LocalDate, IndexedDatapoint> tariffExportDailyMinimumMap =
         IntStream.range(0, tariffExportMovingAverage.size())
+            .filter(i -> tariffExportMovingAverage.get(i).getValue() != null)
             .mapToObj(
                 i -> {
                   ValueDatapoint<?> dp = tariffExportMovingAverage.get(i);
@@ -2447,24 +2449,30 @@ public List<ValueDatapoint<?>> movingAverage(List<ValueDatapoint<?>> dataPoints,
     }
 
     double sum = 0.0;
+    int unavailableCount = 0;
 
     for (int i = 0; i < dataPoints.size(); i++) {
       Double value = (Double) dataPoints.get(i).getValue();
-      sum += value;
+      if (value == null) {
+        unavailableCount++;
+      } else {
+        sum += value;
+      }
 
       if (i >= window) {
         Double valueOld = (Double) dataPoints.get(i - window).getValue();
-        sum -= valueOld;
+        if (valueOld == null) {
+          unavailableCount--;
+        } else {
+          sum -= valueOld;
+        }
       }
 
       if (i >= window - 1) {
-        double avg = sum / window;
-
-        ValueDatapoint<?> original = dataPoints.get(i);
-
-        ValueDatapoint<Double> averaged =
-            new ValueDatapoint<>(original.getTimestamp(), round(avg, 7));
-        result.add(averaged);
+        // Keep unavailable windows in the result so indices still map to the input tariffs.
+        // A partial window must not be used to select a charge or discharge zone.
+        Double avg = unavailableCount == 0 ? round(sum / window, 7) : null;
+        result.add(new ValueDatapoint<>(dataPoints.get(i).getTimestamp(), avg));
       }
     }
 

ems/src/test/groovy/org/openremote/extension/ems/manager/optimisationMethods/EmsTariffNullTest.groovy

@@ -0,0 +1,104 @@
+/*
+ * Copyright 2026, OpenRemote Inc.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ * SPDX-License-Identifier: AGPL-3.0-or-later
+ */
+package org.openremote.extension.ems.manager.optimisationMethods
+
+import org.openremote.model.datapoint.ValueDatapoint
+import spock.lang.Specification
+
+import java.time.LocalDate
+import java.time.ZoneId
+
+class EmsTariffNullTest extends Specification {
+
+  private static final long START = LocalDate.of(2026, 10, 28).atStartOfDay(ZoneId.systemDefault()).toInstant().toEpochMilli()
+
+  def "#implementation.simpleName moving average preserves unavailable windows: #scenario"() {
+    given:
+    def optimisation = implementation.getDeclaredConstructor().newInstance()
+    def input = datapoints(values)
+
+    when:
+    def result = optimisation.movingAverage(input, window)
+
+    then:
+    result.collect { it.value } == expected
+    result.collect { it.timestamp } == input.drop(window - 1).collect { it.timestamp }
+
+    where:
+    [implementation, scenario, values, window, expected] << [EmsOptimisation, EmsOptimisationBeta].collectMany { type ->
+      [
+        [type, "negative and zero prices", [-4, 0, 4, 8], 2, [-2d, 2d, 6d]],
+        [type, "first unavailable", [null, 2, 4, 6], 2, [null, 3d, 5d]],
+        [type, "middle unavailable", [2, 4, null, 6, 8], 2, [3d, null, null, 7d]],
+        [type, "last unavailable", [2, 4, 6, null], 2, [3d, 5d, null]],
+        [type, "consecutive unavailable", [null, null, 2, 4], 2, [null, null, 3d]],
+        [type, "all unavailable", [null, null, null], 2, [null, null]],
+        [type, "single point windows", [2, null, 4], 1, [2d, null, 4d]],
+        [type, "default eight point window", [1, 2, 3, null, 5, 6, 7, 8], 8, [null]],
+        [type, "rounding", [0, 0, 1], 3, [0.3333333d]]
+      ]
+    }
+  }
+
+  def "#implementation.simpleName tariff zones use only complete windows: #scenario"() {
+    given:
+    def optimisation = implementation.getDeclaredConstructor().newInstance()
+    def imports = datapoints(importValues)
+    def exports = datapoints(exportValues)
+    def expected = [:]
+    chargeIndices.each { expected[START + it * 900_000L] = 1 }
+    dischargeIndices.each { expected[START + it * 900_000L] = -1 }
+
+    expect:
+    optimisation.calculateTariffChargeAndDischargeZones(imports, exports, 8) == expected
+
+    where:
+    [implementation, scenario, importValues, exportValues, chargeIndices, dischargeIndices] <<
+    [EmsOptimisation, EmsOptimisationBeta].collectMany { type ->
+      [
+        [
+          type,
+          "valid windows after an import gap",
+          [null] + [-5] * 8 + [100] * 8,
+          [100] * 9 + [0] * 8,
+          (1..8),
+          (9..16)
+        ],
+        [
+          type,
+          "valid windows after an export gap",
+          [100] * 9 + [0] * 8,
+          [null] + [-5] * 8 + [100] * 8,
+          (9..16),
+          (1..8)
+        ],
+        [type, "both unavailable", [null] * 8, [null] * 8, [], []],
+        [type, "import unavailable", [null] * 8, [10] * 8, [], (0..7)],
+        [type, "export unavailable", [10] * 8, [null] * 8, (0..7), []],
+        [type, "no complete window", [10] * 4 + [null] + [10] * 3, [null] * 8, [], []]
+      ]
+    }
+  }
+
+  private static List<ValueDatapoint<?>> datapoints(List values) {
+    values.withIndex().collect { value, index ->
+      new ValueDatapoint<>(START + index * 900_000L, value == null ? null : value.doubleValue())
+    }
+  }
+}

entsoe/README.md

@@ -37,3 +37,38 @@ You'll for instance find that `10YNL----------L` is the code to use for The Neth
 Make sure to also add the `Has predicted data points` configuration to your attribute.
 
 ![](img/ENTSO-E-agent-link.png)
+
+### Unavailable prices and null predictions
+
+**Upgrade note:** Review all forecast consumers for null handling before deploying this change, including custom rules and external integrations. See the [release notes and migration guidance](RELEASE_NOTES.txt).
+
+**An unavailable price is stored as a predicted datapoint with a timestamp and a `null` value. The datapoint entry is not deleted.**
+
+When a point in an otherwise valid returned period has no `price.amount`:
+
+- For `A01`, the agent writes `null` for that point's interval.
+- For `A03`, the agent writes `null` at each resolution interval from that point up to the next point, or the period end if it is the final point. The end boundary is excluded.
+
+For example, if a stored prediction at `12:15` has price `80` and a later publication explicitly marks that interval unavailable, the stored prediction becomes `(12:15, null)`. If no prediction exists at that timestamp, a null-valued entry is inserted. A later publication with an available price can replace the null value.
+
+Reading predicted datapoints with the `all` query retains these timestamped null entries. In JSON responses, an unavailable entry has an `x` timestamp and an omitted `y` property. LTTB downsampling omits unavailable entries entirely, so it does not preserve gap markers. Consumers must treat `null` as **unavailable**, not as zero, and must not carry a previous price forward through those intervals. Code that expects every prediction to contain a number must handle null values explicitly.
+
+Only intervals whose start timestamp is at or after the current time are updated. Historical predictions are left unchanged.
+
+This behavior applies to explicitly present points without a price. An omitted position in `A03` continues the preceding block; it does not by itself mean unavailable. Incomplete `A01` periods are rejected. Fetch failures, no-data acknowledgements, and rejected periods do not clear existing predictions.
+
+A present `price.amount` must contain a valid number. Malformed amounts (including empty or whitespace-only elements) reject the entire period before any of its predictions are updated; they do not mean unavailable. Other valid periods still update their predictions, but gap clearing is disabled for the whole response.
+
+### Gaps between returned periods
+
+When every returned series and period passes validation, the agent also sets existing future predictions in gaps between periods to `null`. These entries retain their timestamps; they are not deleted. Periods are considered together across all returned series, so an interval covered by another series is not a gap.
+
+A gap starts at the preceding covered period's exclusive end and stops at the next covered period's start. Since the gap has no declared resolution, the agent updates only timestamps already stored there, including timestamps off the returned periods' resolution grids. It does not create new predictions in an empty gap. Historical entries and predictions before the first or at/after the last returned period's end are left unchanged.
+
+If any series or period is rejected (including unsupported curve types, malformed or incomplete periods, and empty series or periods), gap clearing is skipped for the whole response. Valid periods still update their own predictions. This prevents rejected data from being mistaken for an unavailable interval.
+
+This gap handling does not remove old timestamps inside a covered period when its resolution changes.
+
+### OpenRemote compatibility
+
+Gap handling requires OpenRemote `1.33.0` or later, which includes JSON-null support in the core datapoint queries. The agent uses the core `AssetDatapointAllQuery` to read existing predictions in gaps, including entries already set to null by an earlier poll. No extension-specific SQL workaround is needed.

entsoe/RELEASE_NOTES.txt

@@ -0,0 +1,39 @@
+Unreleased — ENTSO-E gap handling and EMS tariff availability
+
+BREAKING BEHAVIOUR: predicted prices can contain timestamped null values
+
+ENTSO-E now writes null predictions for explicitly unavailable prices and clears
+existing future predictions in validated gaps between returned periods to null.
+These entries retain their timestamps; they are not deleted. This can replace a
+previously available price. A later publication can restore a numeric price.
+
+This affects every consumer of these forecasts, including custom rules, EMS or
+other extensions, external applications, and integrations outside this repository.
+Consumers that assume every datapoint is numeric may fail during unboxing,
+arithmetic, comparisons, aggregation, or interpolation. Updating OpenRemote core
+alone does not make these consumers null-safe. Review them before deploying this
+ENTSO-E update.
+
+Required consumer adaptations:
+- Treat null as unavailable. Do not interpret it as zero or carry a previous price
+  through an explicitly unavailable interval.
+- Preserve time alignment when handling gaps. Dropping null entries and then
+  treating the remaining entries as consecutive can bridge unavailable intervals
+  or assign decisions to the wrong timestamps.
+- Define how unavailable intervals affect calculations. EMS now excludes any
+  tariff averaging window containing a null and continues with complete windows.
+  Import and export windows are evaluated independently; if neither has a complete
+  window for a day, there are no tariff-based zones for that day. Other optimisation
+  logic continues. Both EmsOptimisation and EmsOptimisationBeta include this fix.
+- Handle absent JSON values as well as explicit nulls: the core 'all' query retains
+  the timestamp as 'x' but omits 'y' for an unavailable value. LTTB omits unavailable
+  points entirely and must not be used to detect or preserve all gap markers.
+- Test unavailable values at the beginning, middle, and end of a forecast, all-null
+  forecasts, repeated polling, and recovery when prices become available again.
+
+Scope and compatibility:
+- OpenRemote 1.33.0 or later is required for the core JSON-null query support.
+- Malformed price amounts reject the entire affected period and disable gap
+  clearing for the response. They are not interpreted as unavailable prices.
+- The EMS fix covers tariff averaging and zone selection in this repository. It
+  does not establish that custom rules or consumers in other projects are safe.