Should actor rootfs be read-only?

[fkautz]

Raised in the review discussion on #184 (#184 (comment)): should sandboxed actor workloads get a read-only root filesystem?

atelet currently builds actor OCI specs with Root.Readonly: false. If we harden this, a few things need working through:

• Workloads that write scratch data (/tmp, /run, app-specific paths) would need writable tmpfs mounts in the spec; which paths to provide by default needs deciding.
• The actor identity directory is unaffected: it is already its own read-only bind mount at /run/ate, independent of rootfs writability.
• Interaction with checkpoint/restore needs checking: gVisor checkpoints include filesystem deltas, and tmpfs contents are part of sentry memory, so the snapshot impact of a read-only rootfs plus tmpfs scratch needs verifying.
• Worth deciding alongside the broader projection design in Add ActorID to the injected actor's process #178, since both shape what the guest filesystem contract looks like.

Counterpoints worth weighing:

• Each actor's rootfs is already private and freshly extracted per restore, behind the sandbox. A read-only rootfs is defense in depth against in-guest tampering and persistence, not host protection, so the win is smaller than in unsandboxed runtimes.
• Kubernetes makes readOnlyRootFilesystem a per-container opt-in because many images expect writable paths. An ActorTemplate field may fit better than changing the default for all workloads.
• The golden-snapshot pattern encourages expensive init before checkpoint, and init that writes to the filesystem is a legitimate form of that. Forcing those writes into tmpfs moves them from filesystem deltas into the memory image, with unverified snapshot-size consequences.

Related: #220 and #232 (working-space and external volume mounts), which a read-only rootfs would turn from convenience into requirement.

[BenTheElder]

The downside here is compatibility with existing images and tools. While we need special handling for capturing durable writes, forcing anything that doesn't mind ephemeral data to move to dedicated paths vs relying on overlay behavior seems a little heavy handed. Right now the whole FS is scratch space, with TBD "homedir" snapshot and external volumes.