Replace CAS with CS and CLS by meroton-benjamin · Pull Request #347 · buildbarn/bb-storage

meroton-benjamin · 2026-06-22T07:14:14Z

This commit builds on top of our split and splice blob support to make
it a mandatory first class feature in Buildbarn. With this commit the
Content Addressable Storage (CAS) is created from two Storage
configurations that work in tandem. A Chunk Storage (CS) which is
content addressed and contains chunks of blobs, and a Chunk List Storage
(CLS) which is addressed by a blob digest and contains a manifest
describing the chunks that make up the blob.

All api calls are automatically translated to use Chunk Lists created
with RepMaxCDC. Effectively this means that large blobs no longer exists
in the storage layer, individual chunks of the large blobs are in turn
deduplicated in such a manner that the chunks are stored only once.

The automatic translation makes certain that clients that are not cdc
aware can still continue to use the storage backend without performing
any changes. Clients which support RepMaxCDC also gets a significant
reduction in the amount of blobs to transfer as they only need to
transfer modified chunks rather than the entire blob.

EdSchouten · 2026-06-22T11:46:20Z

+
+	// Chunk list is marked for validation bypass, push it directy to
+	// downstream blob store.
+	if cdc.ChunkListValidationBypassed(ctx) {


To me the notion of whether a chunk list is known to be valid isn't really a properly of the calling context. It's more of a property of the chunk list that's passed in. Maybe better to either change the signature of BlobAccess.Put(), or add this to buffer.Buffer?

I take it that this logic is added to make sure that if a client performs a legacy Write() for a large object, that the frontend doesn't re-read the objects just to make sure that the ChunkList is valid, right? If so, how does this actually relate to buffer.Source? Maybe we should just treat ChunkLists created by bb-frontend itself as being buffer.BackendProvided(), and use that to skip validation?

EdSchouten · 2026-06-22T11:53:32Z

+			if casConfiguration.ChunkStorage == nil {
+				return status.Error(codes.InvalidArgument, "The Chunk Storage is a mandatory part of the Content Addressable Storage.")
+			}
+			if casConfiguration.ChunkListStorage == nil {
+				return status.Error(codes.InvalidArgument, "The Chunk List Storage is a mandatory part of the Content Addressable Storage.")
+			}


Is this really a necessary requirement? I can imagine that for frontends that implement the full REv2 API it's necessary that both are provided. But for individual shards of our storage backends there is no requirement that each node provides both a CS and CLS.

This concern still holds. :-)

EdSchouten · 2026-07-23T15:51:04Z

+  // an outer ChunkValidatingBlobAccess that has view of the entire
+  // Content Addressable Storage (CAS).
+  //
+  // This option is only supported for the CLS.


This option is only supported for the CS?

EdSchouten · 2026-07-23T15:52:28Z

+
+  // Optional: In memory cache for getting the capabilities for a
+  // specific instance name. Can be set to a relatively low value like
+  // '60s'.


This is only used by the CS?

EdSchouten · 2026-08-18T13:20:25Z

 		grpcClientFactory := grpc.NewBaseClientFactory(grpc.BaseClientDialer, nil, nil, nil)

-		blobAccessCreator := blobstore_configuration.NewCASBlobAccessCreator(
+		blobAccessCreator := blobstore_configuration.NewCSBlobAccessCreator(


Keep in mind that bb_copy can, for example, be used to make backups of actions. This means that for bb_copy it's not sufficient to only use a Chunk Store. It should also preserve/expand chunk lists, etc.

EdSchouten · 2026-08-18T13:22:01Z

@@ -33,7 +33,7 @@ func main() {
 			return util.StatusWrap(err, "Failed to apply global configuration options")


Now that we only deal with chunks and no longer do any streaming, do you think we still need bb_replicator? Maybe it's fine to just throw it out altogether, and let bb_frontend do the fixups directly?

EdSchouten · 2026-08-18T13:22:35Z

+			if casConfiguration.ChunkStorage == nil {
+				return status.Error(codes.InvalidArgument, "The Chunk Storage is a mandatory part of the Content Addressable Storage.")
+			}
+			if casConfiguration.ChunkListStorage == nil {
+				return status.Error(codes.InvalidArgument, "The Chunk List Storage is a mandatory part of the Content Addressable Storage.")
+			}


This concern still holds. :-)

EdSchouten · 2026-08-18T13:29:35Z

+// NewCasChunkingBlobAccess creates a cas blob access configuration that
+// constructs large case objects from the chunks described in the chunk
+// list.
+func NewCasChunkingBlobAccess(chunkStorage, chunkListStorage blobstore.BlobAccess, chunkListFetcher buffer.ChunkListFetcher, maximumMessageSizeBytes int) blobstore.BlobAccess {


This type would also be removed, right?

EdSchouten · 2026-08-18T13:30:38Z

+	if params == nil {
+		return Parameters{}, status.Error(codes.Unimplemented, "This server implementation is only compatible with RepMaxCDC")
+	}
+	if params.MinChunkSizeBytes < 64 {


Just as a safety belt, also require that MaxChunkSizeBytes isn't larger than some big constant? The same thing with the HorizonSizeBytes, which should also be limited. Maybe better to also require a MinChunSizeBytes of more than 64? Seems pretty evil if clients can be tricked into using a ridiculously small chunk size that would cause very high memory usage.

EdSchouten · 2026-08-18T13:34:08Z

+func (ba *chunkListValidatingBlobAccess) matchesStoredChunkList(ctx context.Context, d digest.Digest, userResponse *remoteexecution.SplitBlobResponse) bool {
+	existingMsg, err := ba.BlobAccess.Get(ctx, d).ToProto(&remoteexecution.SplitBlobResponse{}, ba.maximumMessageSizeBytes)
+	if err != nil {
+		return false


This suppresses errors. Is that intentional?

EdSchouten · 2026-08-18T13:41:20Z

+					},
+				}),
+			})
+		}


I think that instead of doing this, we should:

Rename GetDefaultCapabilitiesProvider() to something like GetLeafCapabilitiesProvider(). Or GetBaseCapabilitiesProvider(). That way it's more explicit that it should only be used by 'leaf'/'base' data stores.

Extend that method to take a ChunkingParameters argument.

For the non-CS data stores, let it fail if chunking parameters are provided.

For the CS, do the inverse: require that chunking parameters are given.

EdSchouten · 2026-08-18T13:42:34Z

+// BlobAccessInfo for Content Addressable Storage (CAS) and BlobAccess
+// objects for its constituent blobstores Chunk Storage (CS) and Chunk
+// List Storage (CLS).
+func NewCASBlobAccessFromConfiguration(terminationGroup program.Group, configuration *pb.ContentAddressableStorage, grpcClientFactory grpc.ClientFactory, maximumMessageSizeBytes int, zstdPool bb_zstd.Pool) (contentAddressableStorage BlobAccessInfo, chunkStorage, chunkListStorage blobstore.BlobAccess, err error) {


This also needs to be removed, right?

This commit adds support for the SplitBlob and SpliceBlob methods from the Remote Execution v2 (REv2) api. SplitBlob and SpliceBlob can be used to facilitate uploads and downloads of large files but a naïve implementation like this has some major drawbacks as well. The blobs must exist in both their chunked and non chunked form, which may significantly increase storage requirements for large blobs. The protocol gives no guarantee that a large blob stored in the CAS exists in its chunked form which forces you to perform a fairly heavy Split call that loads the entire large blob in order to decomposition it into its chunks. This implementation mostly exists as a stepping stone for a different implementation where Buildbarn internally manages all blobs as chunked blobs.

This commit builds on top of our split and splice blob support to make it a mandatory first class feature in Buildbarn. With this commit the Content Addressable Storage (CAS) is created from two Storage configurations that work in tandem. A Chunk Storage (CS) which is content addressed and contains chunks of blobs, and a Chunk List Storage (CLS) which is addressed by a blob digest and contains a manifest describing the chunks that make up the blob. All api calls are automatically translated to use Chunk Lists created with RepMaxCDC. Effectively this means that large blobs no longer exists in the storage layer, individual chunks of the large blobs are in turn deduplicated in such a manner that the chunks are stored only once. The automatic translation makes certain that clients that are not cdc aware can still continue to use the storage backend without performing any changes. Clients which support RepMaxCDC also gets a significant reduction in the amount of blobs to transfer as they only need to transfer modified chunks rather than the entire blob.

meroton-benjamin force-pushed the cdc-support-step-2 branch 2 times, most recently from a6748b0 to c2f0bf7 Compare June 22, 2026 10:21

EdSchouten reviewed Jun 22, 2026

View reviewed changes

meroton-benjamin force-pushed the cdc-support-step-2 branch from c2f0bf7 to 6e647c6 Compare July 15, 2026 16:33

meroton-benjamin force-pushed the cdc-support-step-2 branch 2 times, most recently from fe1daa0 to c2608ce Compare August 17, 2026 15:20

EdSchouten reviewed Aug 18, 2026

View reviewed changes

meroton-benjamin added 2 commits August 18, 2026 15:51

meroton-benjamin force-pushed the cdc-support-step-2 branch from c2608ce to 2d6fff1 Compare August 18, 2026 13:52

meroton-benjamin mentioned this pull request Aug 22, 2026

Add ZSTD ReadOffset support #362

Open

		@@ -33,7 +33,7 @@ func main() {
		return util.StatusWrap(err, "Failed to apply global configuration options")

Uh oh!

Conversation

meroton-benjamin commented Jun 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants