[Feature] External volume mounts

[msau42]

Continuing the discussion in #119 (comment) and #220, let's start discussing the use cases and requirements for external volume mounts.

Use Cases

One use case I see discussed in the above issues is the need for each actor to have their own workspace to store data. Rootfs or working dir is insufficient because:

• At larger capacities, there is a performance penalty of checkpointing local directories every time an actor is paused/suspended
• Actor requires higher persistence guarantees with every IO rather than at the pause/suspend timeframe.

Open questions

• Do volumes need to be sharable across multiple actors? If so, as read-only or writeable?

Proposal

Here's my initial thoughts on external volume semantics we can add to substrate.

Lifecycle

External volumes are an independent filesystem mount that follows the Actor lifecycle. A unique volume will be created per actor. Once a volume is created, it is persisted across the lifecycle of the actor. It is deleted when the actor is deleted.

If we support actor updates in the future, the volume will persist across updates. A volume can be added/removed across updates, but an existing volume cannot be mutated (for now). When a volume is deleted from the template, the underlying volume is also deleted after the rollout has completed.

Data Persistence Semantics

External volume mounts are expected to provide filesystem consistency semantics. Writes are persisted when flushed to the backing store. This is different than rootfs semantics where data is only guaranteed to be persisted when the actor is paused/suspended.

When an actor is paused/suspended, we do not take a snapshot of the volume mount (at least in the initial phase). We just unmount the volume and remount it when the actor is resumed. This implies that on resumption, the actor will get the latest volume state, regardless of if the actor gracefully completed its previous task or it crashed prematurely. The actor has to be able to handle the potential difference between its memory state and filesystem state.

In a future milestone, we can explore adding external volume snapshot support on actor suspension, and the ability to create actor volumes from a given volume snapshot.

Storage Backends

The storage backend has to support the following functionality at the required performance / scale:

• Create/delete volumes
• Attach/detach volumes to nodes
• Mount/unmount volumes to nodes
• Security/isolation of volume mounts
• (Future) snapshot of volumes and clone from snapshot

NFS appears to be a commonly requested storage integration for this use case, however, while the file protocol is standard, the control plane APIs are typically vendor-specific. Rather than add a hard-dependency on NFS in substrate, I propose that we focus on supporting Container Storage Interface (CSI), which is well-defined and supported in the Kubernetes ecosystem.

API

kind: ActorTemplate
metadata:
  name: my-agent-template
spec: 
  ...
  containers:
  - image: my-agent-image:v1.2.3
    volumeMounts:
    - name: vol1
      mountPath: /my-data
  volumes:
  - name: vol1
    externalVolumeTemplate:
      capacity: 5Gi
      storageClassName: storage-pool1

[msau42]

chenggui53 (@chenggui53) would love to hear more about your use case and if this proposal could address it.

[chenggui53]

chenggui53 (@chenggui53) would love to hear more about your use case and if this proposal could address it.

Michelle Au (@msau42) Thanks for your support on this problem.

In our primary scenario, we need to maintain many agent instances (e.g., OpenClaw/Hermes) within a substrate actor. Each agent instance must persist its own data when the instance is paused, or when its node crashes – basically any fault that could cause in-memory data loss.

Therefore, I think a persistent volume with the same lifecycle as the actor could solve part of our problem.

As an open question: sometimes we also need a volume that can be shared among actors within a certain scope (e.g., a namespace). Such a shared volume could hold read‑only data maintained by an admin.

So I’d like to ask: is it possible to provide both actor‑lifecycle volumes and shared volume mounts for actors?