Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
63dd2e7
feat(storage): define outbound stage contracts
lykakis Sep 28, 2026
e969471
feat(storage): validate standalone SMS backend profiles
lykakis Sep 28, 2026
78de022
feat(storage): add memory pending and router stores
lykakis Sep 28, 2026
30102dd
feat(storage): add memory routed-work ownership
lykakis Sep 28, 2026
47fbe02
feat(storage): coordinate outbound lifecycle and terminal cleanup
lykakis Sep 29, 2026
05fa118
feat(storage): connect held multipart and protocol admission
lykakis Sep 29, 2026
06ee2dc
feat(storage): connect lifecycle routing and provider completion
lykakis Oct 1, 2026
220f86b
feat(storage): schedule parent-owned provider retries
lykakis Oct 1, 2026
2b5b7f9
test(storage): preserve startup logs under CI log levels
lykakis Oct 1, 2026
c21d17f
feat(storage): drain outbound executions before graceful shutdown
lykakis Oct 5, 2026
f5a7dc9
feat(storage): activate standalone message outbound pipeline
lykakis Oct 6, 2026
d58c173
test(storage): verify combined outbound and embedded lifecycles
lykakis Oct 6, 2026
4fda6b3
refactor(storage): clarify outbound lifecycle API and documentation
lykakis Oct 8, 2026
6451ba3
Merge remote-tracking branch 'origin/main' into feat/338-outbound-sto…
lykakis Oct 8, 2026
f88ef50
test(storage): exclude routing fixture from CDI discovery
lykakis Oct 8, 2026
db7cd63
fix(storage): reserve worker TPS permits only for queued messages
lykakis Oct 8, 2026
c0a0ca6
fix(storage): skip router backlog scans during terminal cleanup
lykakis Oct 8, 2026
bbfaeaa
fix(storage): isolate provider preparation in per-worker pools
lykakis Oct 9, 2026
5e7495f
fix(storage): continue productive outbound batches without delay
lykakis Oct 9, 2026
9a6ea9c
test(storage): address code-quality review annotations
lykakis Oct 9, 2026
310dc5b
fix(storage): restore outbound progress under sustained load
lykakis Oct 20, 2026
bb390a2
Merge remote-tracking branch 'origin/main' into feat/338-outbound-sto…
lykakis Oct 21, 2026
a2538ce
feat(storage): define and round-trip the accepted-source record
lykakis Oct 20, 2026
add8e71
test(storage): make the shared-rate-limit regression deterministic
lykakis Oct 20, 2026
e9ac110
feat(storage): return accepted handles before protocol success
lykakis Oct 20, 2026
68ccd3c
test(storage): prove safe source completion
lykakis Oct 20, 2026
899e3f5
feat(storage): assign accepted parts to distinct multipart groups
lykakis Oct 20, 2026
a5db0d8
feat(storage): reconstruct assembly and expiry from accepted records
lykakis Oct 20, 2026
3624612
docs(storage): document the implemented pending boundary
lykakis Oct 21, 2026
6ab7be4
refactor(storage): make UDH reference bounds explicit
lykakis Oct 21, 2026
6afc5cd
Merge remote-tracking branch 'origin/main' into feat/343-cbor
lykakis Nov 5, 2026
b64d3a0
fix(storage): preserve first-ordinal-wins multipart semantics
lykakis Nov 5, 2026
51976d6
docs(storage): restore numbered lifecycle stage notes
lykakis Nov 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 128 additions & 16 deletions docs/14-outbound-storage.md

Large diffs are not rendered by default.

314 changes: 314 additions & 0 deletions docs/15-pending-record-schema.md

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion docs/DocumentationMap.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Sendium is an open-source, headless SMS gateway for high-throughput messaging. I
| Review current features and roadmap | [12. Features And Roadmap](12-features-roadmap.md) |
| Configure PostgreSQL DLR persistence and review durability | [13. DLR Persistence](13-dlr-persistence.md) |
| Understand outbound storage and extension contracts | [14. Outbound Message Storage](14-outbound-storage.md) |
| Implement pending-source encoding and reconstruction | [15. Pending Record Schema](15-pending-record-schema.md) |
| Contribute code or docs | [Contributing](../.github/CONTRIBUTING.md) |

## Core Concepts
Expand Down Expand Up @@ -65,6 +66,7 @@ Sendium expects these files in the configured `conf` directory. The Docker quick
| [12. Features And Roadmap](12-features-roadmap.md) | Current product capabilities, planned roadmap phases, and related feature documentation. |
| [13. DLR Persistence](13-dlr-persistence.md) | PostgreSQL setup, retention, restart guarantees, and durability limits. |
| [14. Outbound Message Storage](14-outbound-storage.md) | Diagram-led lifecycle guide, class responsibilities, capacity bounds, retries, shutdown, profiles, and extension contracts. |
| [15. Pending Record Schema](15-pending-record-schema.md) | Versioned CBOR source fields, multipart membership/deadlines, supported extensions, validation and record-driven reconstruction boundaries. |
| [Kannel migration converter](https://cytechmobile.github.io/sendium/kannelconverter/) | Browser-only helper for turning a legacy `kannel.conf` into Sendium starter files. |

## API Discovery
Expand All @@ -77,7 +79,7 @@ When Sendium is running, the HTTP API can be inspected through:
| `/swagger-ui` | Interactive Swagger UI. |
| `/openapi.json` | OpenAPI specification. |
| `/q/metrics` | Prometheus-compatible Micrometer metrics endpoint. |
| `/q/health/ready` | Readiness status, including PostgreSQL DLR storage availability. |
| `/q/health/ready` | Readiness status, including configured DLR storage and the standalone outbound pipeline. |

## Community And Project Files

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,11 @@
import gr.cytech.sendium.core.message.StandardMessage;
import gr.cytech.sendium.core.outbound.DefaultOutboundCoordinator;
import gr.cytech.sendium.core.outbound.OutboundCoordinator;
import gr.cytech.sendium.core.outbound.OutboundWork.SourceId;
import gr.cytech.sendium.core.queue.InMemoryQueueProvider;
import gr.cytech.sendium.core.storage.OutboundStage;
import gr.cytech.sendium.core.storage.OutboundStorageException;
import gr.cytech.sendium.core.storage.PendingMessageStore;
import gr.cytech.sendium.core.storage.memory.MemoryPendingMessageStore;
import gr.cytech.sendium.core.storage.memory.MemoryRoutedWorkStore;
import gr.cytech.sendium.core.storage.memory.MemorySelectedRouterStore;
Expand All @@ -19,7 +23,10 @@
import jakarta.inject.Inject;
import jakarta.inject.Singleton;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import java.net.URI;
import java.net.http.HttpClient;
Expand All @@ -32,6 +39,11 @@
import java.util.concurrent.CompletableFuture;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.reset;
import static org.mockito.Mockito.spy;

class MessageLifecycleHttpTest {
@RegisterExtension
Expand All @@ -52,14 +64,25 @@ class MessageLifecycleHttpTest {
@Inject
InMemoryQueueProvider legacy;

@Inject
PendingMessageStore<StandardMessage> pending;

@AfterEach
void clearAdmissionFaults() {
reset(pending);
}

@Test
void httpUsesTheExplicitCoordinatorBeanAndReturnsBackpressureAtCapacity() throws Exception {
void successReferencesAcceptedHandle() throws Exception {
URI requestUri = URI.create(endpoint + "?username=http-user&password=secret&from=Sender&to=306900000001&text=Hello" +
"&dlr-url=https%3A%2F%2Fcallback.test%2Fdlr");
try (HttpClient client = HttpClient.newHttpClient()) {
var request = HttpRequest.newBuilder(requestUri).timeout(Duration.ofSeconds(5)).GET().build();
var accepted = client.send(request, HttpResponse.BodyHandlers.ofString());
assertThat(accepted.statusCode()).isEqualTo(202);
var retained = pending.find(new SourceId(UUID.fromString(accepted.body()))).orElseThrow();
assertThat(retained.serial).isEqualTo(accepted.body());
assertThat(retained.body).isEqualTo("Hello");
var full = client.send(request, HttpResponse.BodyHandlers.ofString());
assertThat(full.statusCode()).isEqualTo(503);
assertThat(legacy.getRouterQueue().isEmpty()).isTrue();
Expand All @@ -75,12 +98,54 @@ void httpUsesTheExplicitCoordinatorBeanAndReturnsBackpressureAtCapacity() throws
}
}

@ParameterizedTest
@ValueSource(strings = {"UNAVAILABLE", "CAPACITY_EXCEEDED", "OWNERSHIP_CONFLICT", "UNSUPPORTED", "INVALID_TRANSITION"})
void rejectsPendingAdmissionFailures(String reasonName) throws Exception {
// Strings cross QuarkusExtensionTest's class-loader boundary; application enum instances do not.
var reason = OutboundStorageException.Reason.valueOf(reasonName);
doThrow(new OutboundStorageException(OutboundStage.Role.PENDING, reason, "private storage details"))
.when(pending).admit(any(), any());
try (HttpClient client = HttpClient.newHttpClient()) {
var request = HttpRequest.newBuilder(URI.create(endpoint +
"?username=http-user&password=secret&from=Sender&to=306900000001&text=Hello"))
.timeout(Duration.ofSeconds(5)).GET().build();
var response = client.send(request, HttpResponse.BodyHandlers.ofString());
int expected = switch (reason) {
case UNSUPPORTED -> 400;
case INVALID_TRANSITION -> 500;
default -> 503;
};
assertThat(response.statusCode()).isEqualTo(expected);
assertThat(response.body()).doesNotContain("private storage details");
assertThat(pending.sourcesAfter(null, 1)).isEmpty();
assertThat(legacy.getRouterQueue().isEmpty()).isTrue();
}
}

@Test
void unconfirmedPendingHandleIsNeverAcknowledged() throws Exception {
doReturn(null).when(pending).admit(any(), any());
try (HttpClient client = HttpClient.newHttpClient()) {
var request = HttpRequest.newBuilder(URI.create(endpoint +
"?username=http-user&password=secret&from=Sender&to=306900000001&text=Hello"))
.timeout(Duration.ofSeconds(5)).GET().build();
assertThat(client.send(request, HttpResponse.BodyHandlers.ofString()).statusCode()).isEqualTo(503);
assertThat(pending.sourcesAfter(null, 1)).isEmpty();
assertThat(legacy.getRouterQueue().isEmpty()).isTrue();
}
}

@Singleton
public static class Beans {
@Produces
@Singleton
OutboundCoordinator<StandardMessage> coordinator() {
var pending = new MemoryPendingMessageStore<StandardMessage>(1, Beans::snapshot);
MemoryPendingMessageStore<StandardMessage> pending() {
return spy(new MemoryPendingMessageStore<StandardMessage>(1, Beans::snapshot));
}

@Produces
@Singleton
OutboundCoordinator<StandardMessage> coordinator(MemoryPendingMessageStore<StandardMessage> pending) {
var router = new MemorySelectedRouterStore<>(pending, 1);
var routed = new MemoryRoutedWorkStore<StandardMessage>(1, Beans::snapshot);
var coordinator = new DefaultOutboundCoordinator<>(pending, router, routed);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,73 @@
package gr.cytech.sendium.app.storage;

import gr.cytech.sendium.core.message.DlrReturnMetadata;
import gr.cytech.sendium.core.message.StandardMessage;
import gr.cytech.sendium.core.outbound.OutboundWork.SourceId;
import gr.cytech.sendium.core.storage.PendingMessageStore;
import gr.cytech.sendium.core.storage.memory.MemoryPendingMessageStore;
import gr.cytech.sendium.core.storage.pending.PendingMessageData;
import gr.cytech.sendium.core.storage.pending.PendingRecordCodec;
import gr.cytech.sendium.core.storage.pending.PendingSourceRecord;
import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

class StandaloneMessageSnapshotTest {
@Test
void normalProtocolRecordPreservesProcessingInputs() {
for (var metadata : List.of(DlrReturnMetadata.http("account", "sender", "recipient", "https://callback.test/dlr"),
DlrReturnMetadata.smpp("account", "system", "sender", "recipient"))) {
var message = new StandardMessage();
SourceId source = new SourceId(UUID.randomUUID());
message.serial = source.value().toString();
message.owner_id = "account";
message.systemId = "system";
message.from = "sender";
message.to = "recipient";
message.body = "Hello";
message.priority = 3;
message.ttl = 120;
message.ddt = 5;
message.dcs = StandardMessage.DCS_16BIT;
message.nextTarget = "provider";
message.field1 = 7;
message.field2 = 8;
message.field3 = 1;
message.field4 = "billing-info";
message.acked = true;
message.dlrReturnMetadata = metadata;
message.tlvs = new HashMap<>(Map.of("0427", "1"));
message.attrs = new HashMap<>(Map.of("request", "original"));

try (PendingMessageStore<StandardMessage> pending = new MemoryPendingMessageStore<>(1, StandaloneMessageSnapshot::copy)) {
pending.open();
SourceId accepted = pending.admit(source, message);
var retained = pending.find(accepted).orElseThrow();
message.field4 = "changed after admission";
message.putAttr("routing-only", "not accepted input");
message.tlvs.clear();
var codec = new PendingRecordCodec();
var record = new PendingSourceRecord(accepted, PendingSourceRecord.Disposition.READY, PendingMessageData.capture(retained));
StandardMessage restored = codec.decode(codec.encode(record)).message().restore();
assertThat(restored).usingRecursiveComparison().isEqualTo(retained);
assertThat(restored.field1).isEqualTo(7);
assertThat(restored.field2).isEqualTo(8);
assertThat(restored.field3).isEqualTo(1);
assertThat(restored.field4).isEqualTo("billing-info");
assertThat(restored.dlrReturnMetadata).isEqualTo(metadata);
assertThat(restored.tlvs).containsEntry("0427", "1");
assertThat(restored.attrs).containsExactlyEntriesOf(Map.of("request", "original"));
}
}
}

@Test
void snapshotDetachesProtocolCollectionsAndMutableAttributeValues() {
var message = new StandardMessage();
Expand Down
4 changes: 4 additions & 0 deletions sendium-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-cbor</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -217,17 +217,20 @@ public Response receiveSms(
if (msg.acked) {
msg.dlrReturnMetadata = DlrReturnMetadata.http(usr, msg.from, msg.to, dlrUrl);
}
String acceptedMessageId;
if (outboundCoordinators.isUnsatisfied()) {
queueProvider.getRouterQueue().enqueue(msg);
acceptedMessageId = msg.serial;
} else {
outboundCoordinators.get().accept(new SourceId(gatewayMessageId), msg);
SourceId accepted = outboundCoordinators.get().accept(new SourceId(gatewayMessageId), msg);
acceptedMessageId = accepted.value().toString();
}
if (MessageTrace.shouldLog(configurationHandler, MessageTrace.EVENT_ACCEPTED)) {
logger.info("message.accepted ingress=http {}", MessageTrace.identifiers(msg));
}

return Response.status(Response.Status.ACCEPTED)
.entity(msg.serial)
.entity(acceptedMessageId)
.build();

} catch (OutboundStorageException e) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import gr.cytech.sendium.core.storage.PendingMessageStore;
import gr.cytech.sendium.core.storage.RoutedWorkStore;
import gr.cytech.sendium.core.storage.SelectedRouterStore;
import gr.cytech.sendium.core.storage.pending.MultipartMembership;

import java.time.Duration;
import java.util.ArrayList;
Expand Down Expand Up @@ -122,32 +123,58 @@ public void beginShutdown() {
}

@Override
public void accept(SourceId source, M message) {
accept(source, message, false);
public SourceId accept(SourceId source, M message) {
return accept(source, message, false);
}

private void accept(SourceId source, M message, boolean held) {
private SourceId accept(SourceId source, M message, boolean held) {
Objects.requireNonNull(source, "source");
Objects.requireNonNull(message, "message");
lock.lock();
try {
requireRunning(Role.PENDING);
if (sourceSelections.containsKey(source)) {
return;
return source;
}
if (held) {
pending.admitHeld(source, message);
} else {
pending.admit(source, message);
SourceId accepted = held ? pending.admitHeld(source, message) : pending.admit(source, message);
if (!source.equals(accepted)) {
throw new OutboundStorageException(Role.PENDING, OutboundStorageException.Reason.OWNERSHIP_CONFLICT,
"Pending admission did not confirm the requested source handle");
}
return accepted;
} finally {
lock.unlock();
}
}

@Override
public void acceptHeld(SourceId source, M message) {
accept(source, message, true);
public SourceId acceptHeld(SourceId source, M message) {
return accept(source, message, true);
}

@Override
public MultipartMembership acceptPart(SourceId source, M message, Duration timeout) {
Objects.requireNonNull(source, "source");
Objects.requireNonNull(message, "message");
Objects.requireNonNull(timeout, "timeout");
lock.lock();
try {
requireRunning(Role.PENDING);
MultipartMembership accepted;
if (sourceSelections.containsKey(source)) {
accepted = pending.findMultipart(source)
.orElseThrow(() -> invalid(Role.PENDING, "Multipart source is no longer retained"));
} else {
accepted = pending.admitPart(source, message, timeout);
}
if (accepted == null || !source.equals(accepted.source())) {
throw new OutboundStorageException(Role.PENDING, OutboundStorageException.Reason.OWNERSHIP_CONFLICT,
"Pending admission did not confirm multipart source membership");
}
return accepted;
} finally {
lock.unlock();
}
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import gr.cytech.sendium.core.outbound.OutboundWork.SelectionId;
import gr.cytech.sendium.core.outbound.OutboundWork.SourceId;
import gr.cytech.sendium.core.outbound.OutboundWork.WorkId;
import gr.cytech.sendium.core.storage.pending.MultipartMembership;

import java.time.Duration;
import java.util.Optional;
Expand Down Expand Up @@ -35,11 +36,14 @@ public interface OutboundCoordinator<M extends StandardMessage> extends AutoClos
*/
void beginShutdown();

/** Accepts source ownership and makes the message ready for selection according to the store policy. */
void accept(SourceId source, M message);
/** Accepts ready source ownership and returns the same confirmed handle before protocol success; null is never acceptance. */
SourceId accept(SourceId source, M message);

/** Accepts source ownership without making this source routable yet. */
void acceptHeld(SourceId source, M message);
/** Accepts ownership without making the source routable; returns the same confirmed handle as accept. */
SourceId acceptHeld(SourceId source, M message);

/** Accepts a multipart source with stable group membership before protocol success. */
MultipartMembership acceptPart(SourceId source, M message, Duration timeout);

/**
* Makes already-accepted held sources available to bounded selection using one prepared execution message.
Expand Down
Loading
Loading