Skip to content

feat(storage): define pending-message contracts and CBOR envelope - #355

Draft
lykakis wants to merge 30 commits into
feat/338-outbound-storage-abstractionsfrom
feat/343-cbor
Draft

feat(storage): define pending-message contracts and CBOR envelope#355
lykakis wants to merge 30 commits into
feat/338-outbound-storage-abstractionsfrom
feat/343-cbor

Conversation

@lykakis

@lykakis lykakis commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Stack and review base

Depends on #351 (#338). This draft intentionally targets feat/338-outbound-storage-abstractions, so its diff contains only the #343 follow-up.

Both branches include the latest fetched origin/main. #338's performance fixes are retained. After #351 merges, rebase/retarget this PR onto main and review the resulting diff before merging.

Summary

  • Refine pending admission to return confirmed stable source handles before HTTP/SMPP success; expose backend-neutral lookup and bounded handle traversal.
  • Define the explicit versioned Jackson CBOR accepted-source schema, preserving message fields, DLR metadata, supported typed extensions, multipart assignments and absolute deadlines.
  • Admit multipart parts atomically into distinct group instances. Each new client submission remains independently accepted, including repeated ordinals; internal same-source-ID retries remain idempotent.
  • Reconstruct runtime assembly/expiry from retained accepted records through bounded reads, with explicit corruption/readiness failures and in-process publication retry safety.
  • Prove safe completion through terminal processing and required DLR handoff, including uncertain absence, partial cleanup and lost completion acknowledgements.
  • Document the source schema, lifecycle, compatibility and integration boundaries.

Agreed behavior and scope

  • The clarified repeated-part rule supersedes the original first-ordinal-wins duplicate wording in feat(storage): define the outbound pending-message contract and CBOR envelope #343: for A=part1, B=part1, C=part2, D=part3, assemble A+C+D and retain B separately for its own group/expiry.
  • No durable checkpoint of runtime assembly/publication/release decisions is introduced. Pending-only recovery may repeat processing; provider submission remains at least once.
  • The standalone profile remains memory/memory/memory. Physical filesystem persistence, durable startup/profile enablement and actual forced-restart guarantees remain with feat(storage): implement filesystem pending-message store and ingress #339 and subsequent stage work.
  • Existing memory snapshot/custom-message extension boundaries remain intact; arbitrary custom before-insert filter object preservation is deferred. DLR storage stays independently owned.

Verification

  • mvn -B -ntp clean verify using installed Maven 3.9.14 / Java 25: 525 core unit tests, 39 core integration tests, 60 app tests, zero failures/errors/skips; Checkstyle and packaging pass.
  • npm ci followed by npm run check in sendium-kannel-converter: converter tests and production build pass after integrating main's dependency updates.
  • Documentation checks: 109 local links/anchors, all 54 message-field capture/restore mappings, six DLR return components and five CBOR envelope keys match the implementation.
  • All seven topic commits retain DCO sign-offs and valid cryptographic signatures. Local skill/configuration/design files are not included.

Refs #343
Parent epic: #337

@lykakis
feat(storage): define outbound stage contracts
63dd2e7 
Task 1.1 - Define stage contracts, ownership transitions, and the composition boundary.

Add single-destination lifecycle interfaces, typed work identities, and embedding contracts. Document stage-consistent recovery and verify public contract boundaries.

Refs #338

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): validate standalone SMS backend profiles
e969471 
Task 1.2 - Add standalone backend selection, startup validation, and profile logging.

Reject unavailable profiles before worker startup, preserve core embedding independence, and verify configuration and startup behavior.

Refs #338

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): add memory pending and router stores
78de022 
Task 2.1 - Implement memory pending ownership and bounded router selection.

Retain accepted sources through routing, select indexed eligible work within bounded queued and in-flight capacity, and preserve retry ownership through failures.

Refs #338

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): add memory routed-work ownership
30102dd 
Task 2.1a - Implement single-destination memory routed-work storage.

Track destination takes, retry-safe forwarding, and terminal cleanup while retaining source ownership and rejecting conflicting assignments.

Refs #338

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): coordinate outbound lifecycle and terminal cleanup
47fbe02 
Task 2.1b - Implement the concrete outbound lifecycle coordinator.

Coordinate owned routing, required handoff, retryable cleanup, and shutdown. Remove downstream-only worker forwarding and document scope ownership rules.

Refs #338

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): connect held multipart and protocol admission
05fa118 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): connect lifecycle routing and provider completion
06ee2dc 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): schedule parent-owned provider retries
220f86b 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): preserve startup logs under CI log levels
2b5b7f9 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): drain outbound executions before graceful shutdown
c21d17f 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): activate standalone message outbound pipeline
f5a7dc9 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): verify combined outbound and embedded lifecycles
d58c173 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
refactor(storage): clarify outbound lifecycle API and documentation
4fda6b3 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
Merge remote-tracking branch 'origin/main' into feat/338-outbound-sto…
6451ba3 
…rage-abstractions

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): exclude routing fixture from CDI discovery
f88ef50 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
fix(storage): reserve worker TPS permits only for queued messages
db7cd63 
Empty legacy worker loops acquired rate-limit permits before checking their queues. With lifecycle traffic sharing the same worker limiter, idle polling consumed send capacity and delayed actual submissions.

Acquire a permit only after obtaining a message, preserving one shared per-worker TPS budget for legacy and coordinated attempts. Avoid rate waiting after shutdown and retain the post-wait check that restores taken work.

Verify idle polling, shutdown during dequeue and rate acquisition, and real shared-limiter pacing across legacy and lifecycle sends. The focused SMPP, dispatch and shutdown suite passes 48 tests.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
fix(storage): skip router backlog scans during terminal cleanup
c0a0ca6 
Taken and routed selections have already left the ready deque. Unconditional removal searched the unrelated router backlog on every provider completion while holding shared storage and coordinator locks.

Remove from the ready deque only for queued selections, retaining source-first cleanup, routing capacity accounting and idempotence. Verify queued, taken and routed cleanup preserves unrelated work ordering and capacity; all 80 focused storage and coordinator tests pass.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
fix(storage): isolate provider preparation in per-worker pools
bbfaeaa 
Blocking rate waits, worker filters and SMPP sends occupied the shared two-thread scheduler, allowing a slow destination to delay healthy destinations and limiting effective sending concurrency.

Reserve worker slots before submitting preparation to bounded dispatcher-owned pools sized by threadCount. Preserve the shared per-worker TPS limiter and parent-owned provider retries and handoffs. Retire stopped-worker pools and shut down owned pools after drain without closing application-owned timer schedulers.

Verify destination and timer isolation, three occupied slots with a fourth attempt waiting, pool shutdown and worker replacement. Full reactor verify passes 393 core unit tests, 39 integration tests and 43 app tests.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
fix(storage): continue productive outbound batches without delay
5e7495f 
A mandatory 100 ms pause after every bounded processing cycle imposed a global staging ceiling even when ready messages and provider capacity were available.

Schedule the next bounded cycle immediately after successful routing or provider scheduling. Retain idle, pause and failure backoff so unroutable work cannot trigger a busy loop. Fence queued cycles and rescheduling during shutdown, preserving per-worker TPS and execution limits.

Add seven deterministic scheduling cases and retain real HTTP pipeline and shutdown coverage. Full reactor verify passes 393 core unit tests, 39 integration tests and 50 app tests with zero Checkstyle violations.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): address code-quality review annotations
9a6ea9c 
Match the credential fixture's synchronized getter contract and annotate multipart listener overrides. Explain why CDI observer parameters and the validated-profile producer dependency remain necessary despite unused payload values.

Preserve lifecycle ordering and the existing immutable source snapshots rather than removing dependencies or adding redundant copies to silence analyzer comments. All 42 focused ownership, HTTP and startup-profile tests pass with zero Checkstyle violations.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
fix(storage): restore outbound progress under sustained load
310dc5b 
Reuse private completion monitors and notification-based drain waits to avoid retaining future copies during backlog polling. Drain per-worker FIFO attempts directly instead of scheduling per-message contention timers, preserving preparation concurrency and asynchronous SMPP windows.

Report fatal processing and rescheduling failures through standalone readiness. Cover monitor retention, FIFO and pause scheduling, shutdown drain, and processor failure handling.

Full reactor verify passes 396 core unit tests, 39 integration tests and 53 app tests with zero Checkstyle violations. Matched 100-msg/s HTTP+DLR and SMPP+DLR rehearsals each complete all 60,000 messages and receipts without rejection or unfinished work.

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
Merge remote-tracking branch 'origin/main' into feat/338-outbound-sto…
bb390a2 
…rage-abstractions

Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): define and round-trip the accepted-source record
a2538ce 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): make the shared-rate-limit regression deterministic
add8e71 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): return accepted handles before protocol success
e9ac110 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
test(storage): prove safe source completion
68ccd3c 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): assign accepted parts to distinct multipart groups
899e3f5 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
feat(storage): reconstruct assembly and expiry from accepted records
a5db0d8 
Signed-off-by: pavlos <pavlos@cytech.gr>
@lykakis
docs(storage): document the implemented pending boundary
3624612 
Signed-off-by: pavlos <pavlos@cytech.gr>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 21, 2026
@lykakis
lykakis added this pull request to stack #356 October 21, 2026 10:23
github-code-quality[bot]
github-code-quality Bot found potential problems Oct 21, 2026
View reviewed changes
Comment thread sendium-core/src/main/java/gr/cytech/sendium/util/MessageUtil.java Fixed
Comment thread sendium-core/src/main/java/gr/cytech/sendium/util/MessageUtil.java Fixed
@lykakis
refactor(storage): make UDH reference bounds explicit
6ab7be4 
Signed-off-by: pavlos <pavlos@cytech.gr>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-code-quality github-code-quality[bot] github-code-quality[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lykakis