Skip to content

Add durable asset storage contracts - #1957

Open
arnabnandy7 wants to merge 1 commit into
embabel:mainfrom
arnabnandy7:feature/durable-asset-storage
Open

arnabnandy7 wants to merge 1 commit into
embabel:mainfrom
arnabnandy7:feature/durable-asset-storage

Conversation

@arnabnandy7

Copy link
Copy Markdown
Collaborator

Summary

Adds the repository-side foundation for durable conversation assets.

  • Introduces MaterializableAsset for assets whose content can be copied before their original location expires.
  • Introduces DurableAsset as portable metadata containing an opaque storage URI and content hash.
  • Introduces the AssetStore SPI for storing, opening, and deleting durable asset content.
  • Adds AssetTracker helpers that materialize tool-returned assets before tracking them.
  • Makes ScriptArtifact a materializable, ephemeral asset so sandbox-produced files can participate in the standard asset flow.
  • Documents the distinction between in-memory tracking and durable storage, including retention responsibilities.

The implementation reuses AssetAddingTool for tool-result conversion and tracking.

Motivation

AssistantMessage and Conversation can track assets, but tracking alone does not preserve temporary files or restore their metadata after a process restart.

Script artifacts are especially affected because their paths refer to temporary execution directories. Applications currently need custom plumbing to copy those files before cleanup.

This change establishes the core contracts and capture flow required for durable storage. Conversation-store persistence and rehydration will be handled separately in embabel-chat-store.

Example

val durableTool = conversation.assetTracker
    .addDurablyReturnedAssets(generatingTool, assetStore)

When the wrapped tool returns a MaterializableAsset, its content is stored first and the resulting DurableAsset is added to the tracker.

Verification

  • AssetStoreTest: 2 tests passed
  • ScriptArtifactTest: 2 tests passed

Follow-up

Update embabel-chat-store to persist DurableAsset metadata with assistant messages and restore those assets when conversations are loaded.

Closes #1821

@igordayen igordayen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arnabnandy7 - few comments to consider, thank you

Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt Outdated
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetTracker.kt
Comment thread embabel-agent-docs/src/main/asciidoc/reference/chatbots/page.adoc Outdated
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt
Comment thread embabel-agent-api/src/main/kotlin/com/embabel/chat/AssetStore.kt
Signed-off-by: Arnab Nandy <arnab_nandy7@yahoo.com>
@arnabnandy7
arnabnandy7 force-pushed the feature/durable-asset-storage branch from 2c0d942 to 11bd812 Compare September 7, 2026 17:57
@arnabnandy7
arnabnandy7 marked this pull request as ready for review September 7, 2026 18:01
@igordayen

Copy link
Copy Markdown
Contributor

@arnabnandy7, thanks for addressing inquiries.
From CLAUDE:

  • Line 33 ("can be derived?") — this is about contentHash in DurableAsset. The hash is derived during store()
    — FileSystemAssetStore computes it on the way through DigestInputStream. It's a field on DurableAsset
    because it's persisted metadata (for integrity verification on open()/after reload), not recomputed each
    time. The question is valid: could DurableAsset expose it as a lazy field? But since DurableAsset is a data
    class reconstructed from storage (chat-store PR Examples showing core functionality #19 maps it back), the field needs to be in the constructor.
    No change needed, but worth adding a KDoc line: "Computed by the store during materialization; can be used to
    verify content integrity on retrieval."

  • Line 99 (AssetTracker.inMemory()) — the comment "what about durable, non-in-memory" is a follow-up: you'd
    want an AssetTracker that, on reload, queries the store for assets by session. Not in scope here but the PR's
    follow-up section calls this out explicitly ("Conversation-store persistence and rehydration handled
    separately in embabel-chat-store"). Chat-store PR Examples showing core functionality #19 handles the persistence side. The AssetTracker factory
    is orthogonal — the tracker just holds what the store returns on reload.

However, align these two things:

  1. SPI naming convention — AssetStore uses store/open/delete; AgentProcessSnapshotStore uses
    save/load/delete. Consider aligning verbs: both could use save/load/delete (what AgentProcessSnapshotStore
    already has). Minor, and store as a verb on AssetStore reads naturally for binary blobs.
  2. Autoconfiguration model — AgentProcessSnapshotStore has AgentProcessPersistenceProperties(enabled=false).
    AssetStore has no equivalent Spring Boot conditional. If no AssetStore bean is present, tools wrapping with
    addDurablyReturnedAssets would NPE at runtime. This is the actual gap: AssetStore needs an
    optional/conditional wiring model the same way snapshot store does — either a no-op NoOpAssetStore default,
    or a conditional check before wrapping.
  3. The storageUri in DurableAsset is coupled to the store instance that created it — same design as
    AgentProcessSnapshotStore where the payload is opaque. Consistent. Good.

Recommendation for PR author: Add a NoOpAssetStore or document that addDurablyReturnedAssets must only be
called when a store bean is configured. Without this, the opt-in story is inconsistent with how snapshot
persistence works.

And more....

Package: AssetStore doesn't belong in com.embabel.chat root

The root com.embabel.chat package is domain types — Asset, Conversation, Message, AssetTracker, etc.
AssetStore is an application-supplied SPI (the KDoc even says so). The agent side puts SPIs in
com.embabel.agent.spi.persistence, keeping them separate from core domain.

The PR adds AssetStore to the root domain package, mixing an extension point with domain types. Two options
that match existing conventions:

  • com.embabel.chat.spi — introduce the subpackage to mirror agent.spi
  • com.embabel.chat.store — a chat-centric name, since store is the noun used throughout (chat-store,
    AssetStore)

MaterializableAsset and DurableAsset are domain types (they extend Asset) and belong in the root. Only the
AssetStore interface — the extension point — should move.

Visibility: FileSystemAssetStore should be internal

All support implementations (InMemoryAssetTracker, InMemoryConversation, EventPublishingConversation) are
public in support. No internal is used anywhere — so the current PR is consistent with the convention.

But FileSystemAssetStore is different in one way: it's an opinionated default implementation that ships
inside embabel-agent-api (no external deps so that's fine), but it's also the only concrete AssetStore in the
whole module. Making it internal would force consumers to either implement their own or depend on a separate
infrastructure module — which is arguably the right design (embabel-agent-jcache doesn't expose its cache
provider as public API either).

Summary of what to raise in the review:

  1. Move AssetStore interface to com.embabel.chat.spi (or com.embabel.chat.store) — domain types and SPIs
    mixed in root
  2. Consider internal on FileSystemAssetStore or move it to an optional embabel-agent-filesystem module,
    consistent with how JCache is separated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No durable storage or restoration for Assets produced during a conversation

2 participants