Add VectorStore.upsert for pre-computed embeddings

spring-ai-commons/src/main/java/org/springframework/ai/document/DocumentMetadata.java

@@ -32,7 +32,36 @@ public enum DocumentMetadata {
 	 * The lower the distance, the more they are similar.
 	 * It's the opposite of the similarity score.
 	 */
-	DISTANCE("distance");
+	DISTANCE("distance"),
+
+	/**
+	 * Metadata key holding a pointer to content kept outside the vector store.
+	 * <p>
+	 * Use it when a row's embedding was computed from something that isn't stored
+	 * in the row itself &mdash; an image, a video frame, an audio clip, or a
+	 * document too large to inline. The vector is stored and stays searchable, but
+	 * in place of the content the row keeps a reference to where the content really
+	 * lives (for example an S3 URI, a CDN URL, or a database key). The store treats
+	 * this reference as an opaque string and never resolves it; after a search
+	 * returns the row, the application follows the pointer to fetch the content.
+	 * <p>
+	 * Such a row is created with an empty-text {@link Document} plus this key, and
+	 * written with a caller-supplied embedding through {@code VectorStore.upsert}
+	 * (which stores the vector as given and never embeds). Note the empty text only
+	 * exists to satisfy the document's text-or-media rule; it is not content, so a
+	 * document carrying this key is a reference row even though {@code isText()}
+	 * reports true.
+	 * <p>
+	 * Nothing in the framework reads or validates this key. It is an ordinary
+	 * metadata entry that the application writes and later resolves, and keeping the
+	 * pointer non-empty is up to the caller. A row that still has real text of its own
+	 * can be written with {@code VectorStore.add}, which is how a summary or a chunk
+	 * keeps a pointer back to the full article. A row with empty text has to go
+	 * through {@code upsert}, because {@code add} embeds the text and there is nothing
+	 * there to embed.
+	 * @since 2.1.0
+	 */
+	CONTENT_REF("content_ref");
 
 	private final String value;
 

spring-ai-commons/src/test/java/org/springframework/ai/document/DocumentMetadataTests.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2023-present the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.ai.document;
+
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+class DocumentMetadataTests {
+
+	@Test
+	void distanceKey() {
+		assertThat(DocumentMetadata.DISTANCE.value()).isEqualTo("distance");
+		assertThat(DocumentMetadata.DISTANCE).hasToString("distance");
+	}
+
+	@Test
+	void contentRefKey() {
+		assertThat(DocumentMetadata.CONTENT_REF.value()).isEqualTo("content_ref");
+		assertThat(DocumentMetadata.CONTENT_REF).hasToString("content_ref");
+	}
+
+}

spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs.adoc

@@ -57,6 +57,8 @@ public interface VectorStore extends DocumentWriter, VectorStoreRetriever {
 
     void add(List<Document> documents);
 
+    default void upsert(List<EmbeddedDocument> entries) { ... }
+
     void delete(List<String> idList);
 
     void delete(Filter.Expression filterExpression);
@@ -70,6 +72,7 @@ public interface VectorStore extends DocumentWriter, VectorStoreRetriever {
 ----
 
 The `VectorStore` interface combines both read and write operations, allowing you to add, delete, and search for documents in a vector database.
+While `add` computes the embeddings for you, `upsert` writes documents together with embeddings you already have; see <<writing-precomputed-embeddings,Writing Pre-computed Embeddings>>.
 
 === SearchRequest Builder
 
@@ -437,6 +440,120 @@ void load(String sourceFile) {
 }
 ----
 
+[[writing-precomputed-embeddings]]
+=== Writing Pre-computed Embeddings
+
+`add` computes an embedding for every document inside the store, using the store's embedding model.
+Sometimes you already have the vectors and want to write them as-is: a provider's batch API computed them overnight at half price, another team owns the embedding pipeline, or you are migrating from a system that exports text and vectors together.
+For those cases the `VectorStore` interface offers `upsert`, which takes the document and its vector paired together in an `EmbeddedDocument` and never embeds anything itself:
+
+[source,java]
+----
+public record EmbeddedDocument(Document document, float[] embedding) {
+}
+----
+
+As the name says, `upsert` replaces by id: writing the same id again overwrites the existing row rather than adding a duplicate.
+That makes a re-run with stable ids safe, which is useful for an ingestion job that may be retried after a failure.
+
+[source,java]
+----
+@Autowired
+VectorStore vectorStore;
+
+// a vector you computed elsewhere
+float[] embedding = ...;
+Document document = new Document("8f14e45f-ceea-467a-9a3e-5b1c2d6f7a90", "some text",
+        Map.of("source", "manual"));
+
+this.vectorStore.upsert(List.of(new EmbeddedDocument(document, embedding)));
+----
+
+NOTE: Some stores constrain the document id.
+Qdrant requires a UUID, and `PgVectorStore` stores the id in a UUID column unless you configure `idType`, so an arbitrary string fails on those.
+Deriving the UUID from a key of your own, for example `UUID.nameUUIDFromBytes(key.getBytes(UTF_8))`, keeps the id stable across runs, which is what makes a re-run replace the row rather than add a second one.
+
+To write plain text through `upsert`, embed it first and hand over the result:
+
+[source,java]
+----
+@Autowired
+EmbeddingModel embeddingModel;
+
+float[] vector = this.embeddingModel.embed(document);
+this.vectorStore.upsert(List.of(new EmbeddedDocument(document, vector)));
+----
+
+`upsert` is opt-in per store: the default implementation throws `UnsupportedOperationException`, so a store supports it only once it has been implemented.
+It is available on pgvector, Redis, Elasticsearch, and Qdrant, with more stores to follow.
+When a store knows its index dimension, it checks every vector in the batch against it before writing anything, so a wrong-sized vector fails fast and cannot partially write.
+Where that dimension can only be guessed, the check is left to the database, which rejects the vector with its own error.
+
+Because the vectors come from outside the store, lining them up with the index is yours to get right.
+Three things have to agree, and only the first is checked for you:
+
+Width:: Your embedding model decides the width, and the index has to be created with that same width, for example `dimensions(1024)` on the store builder or a pre-created index whose mapping says 1024.
+A mismatch fails on the first write.
+Vector space:: Nothing records which model produced a row, so vectors written by one model and searched with another come back plausible but meaningless, with no error anywhere.
+Writing the model name and version into metadata costs nothing and tells a later reader what a row actually holds.
+The query side:: `similaritySearch` still embeds the query with the store's own `EmbeddingModel`, so that model has to produce vectors in the same space and of the same width as the ones you upserted.
+
+=== Indexing Non-Text or Oversized Content
+
+With text, the store keeps the document and its vector together in one row: search matches on the vector and hands back the text.
+A multimodal model breaks that assumption. It gives you a searchable vector for an image, a video segment, or an audio clip, but the content itself lives in S3, a CDN, or a database, and there is no text to put in the row.
+The same applies to text that is simply too large for the row, such as a full article whose chunks were indexed.
+
+The row then keeps everything except the content: id, metadata, vector, and a pointer to where the content lives.
+The pointer goes in metadata under `DocumentMetadata.CONTENT_REF`, and the store never interprets it: it is an S3 URI, a CDN URL, or a database key that only your application resolves.
+Such a row is written with `upsert`, because you supply the vector from the multimodal model yourself:
+
+[source,java]
+----
+byte[] frame = ...;                                 // the image/video/audio bytes
+float[] embedding = multimodalModel.embed(frame);   // your multimodal model
+
+String key = "video-42/frame-4711";
+
+Document reference = Document.builder()
+    // Stable and UUID-shaped, so a re-run replaces this row
+    .id(UUID.nameUUIDFromBytes(key.getBytes(UTF_8)).toString())
+    .text("")                                       // empty: the content lives elsewhere
+    .metadata(DocumentMetadata.CONTENT_REF.value(), "s3://media/video-42.mp4#t=4711")
+    .metadata("camera", "north-gate")               // still filterable like any row
+    .build();
+
+this.vectorStore.upsert(List.of(new EmbeddedDocument(reference, embedding)));
+----
+
+NOTE: The empty text is what makes the document valid; it is not content. A document carrying `CONTENT_REF` is a reference row even though `isText()` reports true. `upsert` requires text for this reason: no store can persist a row that carries none, so a media document is rejected rather than half-written.
+
+The pointer is an ordinary metadata entry.
+Nothing in the framework reads or validates it, so keeping it non-empty is up to you.
+A row that still has real text of its own can be written with `add`, which is how a summary or a chunk keeps a pointer back to the full article.
+A row with empty text has to go through `upsert`, because `add` embeds the text and there is nothing there to embed.
+
+Search is unchanged. `similaritySearch` matches on vectors and returns `Document` s as always, and for these rows the caller follows the pointer:
+
+[source,java]
+----
+List<Document> results = this.vectorStore.similaritySearch(SearchRequest.builder()
+    .query("a truck at the north gate after dark")   // embedded by the same multimodal model
+    .filterExpression("camera == 'north-gate'")
+    .build());
+
+for (Document doc : results) {
+    String ref = (String) doc.getMetadata().get(DocumentMetadata.CONTENT_REF.value());
+    byte[] frame = mediaStorage.fetch(ref);           // S3, CDN, wherever the pointer leads
+}
+----
+
+Text queries against image vectors work when the store's embedding model is the multimodal model's text encoder, so query and content land in the same vector space.
+
+NOTE: Redis returns only the metadata fields declared when the store is built, so `content_ref` has to be declared for the pointer to survive a read: `.metadataFields(MetadataField.tag(DocumentMetadata.CONTENT_REF.value()))`.
+The same goes for any field you filter on, such as `camera` above.
+Adding a field to a store that already has an index means recreating the index.
+
 === Reading from a Vector Store
 
 Later, when a user question is passed into the AI model, a similarity search is done to retrieve similar documents, which are then "stuffed" into the prompt as context for the user's question.

spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/elasticsearch.adoc

@@ -214,6 +214,12 @@ is converted into the proprietary Elasticsearch filter format:
 (metadata.author:john OR jill) AND metadata.article_type:blog
 ----
 
+== Upsert support
+
+`ElasticsearchVectorStore` implements `upsert(List<EmbeddedDocument>)`, so you can write documents together with embeddings you computed elsewhere instead of having the store embed them.
+Writing the same id again replaces the existing row, so a re-run with stable ids does not create duplicates.
+See xref:api/vectordbs.adoc#writing-precomputed-embeddings[Writing Pre-computed Embeddings] for details.
+
 == Manual Configuration
 
 Instead of using the Spring Boot auto-configuration, you can manually configure the Elasticsearch vector store. For this you need to add the `spring-ai-elasticsearch-store` to your project:

spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/pgvector.adoc

@@ -192,6 +192,12 @@ vectorStore.similaritySearch(SearchRequest.builder()
 
 NOTE: These filter expressions are converted into PostgreSQL JSON path expressions for efficient metadata filtering.
 
+== Upsert support
+
+`PgVectorStore` implements `upsert(List<EmbeddedDocument>)`, so you can write documents together with embeddings you computed elsewhere instead of having the store embed them.
+Writing the same id again replaces the existing row, so a re-run with stable ids does not create duplicates.
+See xref:api/vectordbs.adoc#writing-precomputed-embeddings[Writing Pre-computed Embeddings] for details.
+
 == Manual Configuration
 
 Instead of using the Spring Boot auto-configuration, you can manually configure the `PgVectorStore`.

spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/qdrant.adoc

@@ -108,6 +108,12 @@ Properties starting with `spring.ai.vectorstore.qdrant.*` are used to configure
 |`spring.ai.vectorstore.qdrant.initialize-schema`| Whether to initialize the schema | `false`
 |===
 
+== Upsert support
+
+`QdrantVectorStore` implements `upsert(List<EmbeddedDocument>)`, so you can write documents together with embeddings you computed elsewhere instead of having the store embed them.
+Writing the same id again replaces the existing row, so a re-run with stable ids does not create duplicates.
+See xref:api/vectordbs.adoc#writing-precomputed-embeddings[Writing Pre-computed Embeddings] for details.
+
 == Manual Configuration
 
 Instead of using the Spring Boot auto-configuration, you can manually configure the Qdrant vector store. For this you need to add the `spring-ai-qdrant-store` to your project:

spring-ai-docs/src/main/antora/modules/ROOT/pages/api/vectordbs/redis.adoc

@@ -187,6 +187,26 @@ is converted into the proprietary Redis filter format:
 @country:{UK | NL} @year:[2020 inf]
 ----
 
+== Upsert support
+
+`RedisVectorStore` implements `upsert(List<EmbeddedDocument>)`, so you can write documents together with embeddings you computed elsewhere instead of having the store embed them.
+Writing the same id again replaces the existing row, so a re-run with stable ids does not create duplicates.
+See xref:api/vectordbs.adoc#writing-precomputed-embeddings[Writing Pre-computed Embeddings] for details.
+
+Redis stores every metadata key you write, but a search returns only the fields declared through `metadataFields`, so anything else comes back missing.
+Declare the keys you need returned, not just the ones you filter on:
+
+[source,java]
+----
+RedisVectorStore.builder(jedisClient, embeddingModel)
+    .metadataFields(MetadataField.tag(DocumentMetadata.CONTENT_REF.value()))
+    .initializeSchema(true)
+    .build();
+----
+
+The store logs a warning the first time it writes a document carrying an undeclared field.
+Adding a field to a store that already has an index means recreating the index.
+
 == Manual Configuration
 
 Instead of using the Spring Boot auto-configuration, you can manually configure the Redis vector store. For this you need to add the `spring-ai-redis-store` to your project:

spring-ai-docs/src/main/antora/modules/ROOT/pages/observability/index.adoc

@@ -272,14 +272,14 @@ WARNING: If you enable logging of the image prompt data, there's a risk of expos
 All vector store implementations in Spring AI are instrumented to provide metrics and distributed tracing data through Micrometer.
 
 The `db.vector.client.operation` observations are recorded when interacting with the Vector Store. 
-They measure the time spent on the `query`, `add` and `remove` operations and propagate the related tracing information.
+They measure the time spent on the `query`, `add`, `upsert` and `delete` operations and propagate the related tracing information.
 
 .Low Cardinality Keys
 [cols="a,a", stripes=even]
 |===
 |Name | Description
 
-|`db.operation.name` | The name of the operation or command being executed. One of `add`, `delete`, or `query`.
+|`db.operation.name` | The name of the operation or command being executed. One of `add`, `upsert`, `delete`, or `query`.
 |`db.system` | The database management system (DBMS) product as identified by the client instrumentation. One of `pg_vector`, `azure`, `cassandra`, `chroma`, `elasticsearch`, `milvus`, `neo4j`, `opensearch`, `qdrant`, `redis`, `typesense`, `weaviate`, `pinecone`, `oracle`, `mongodb`, `gemfire`, `simple`.
 |`spring.ai.kind` | The kind of framework API in Spring AI: `vector_store`.
 |===
@@ -440,7 +440,7 @@ The following shows how base metric names expand to Prometheus time series.
 |`db_vector_client_operation_seconds_sum`
 |Timer
 |seconds
-|Total time spent in vector store operations (add/delete/query)
+|Total time spent in vector store operations (add/upsert/delete/query)
 
 |`db_vector_client_operation_seconds_count`
 |Counter
@@ -463,7 +463,7 @@ The following shows how base metric names expand to Prometheus time series.
 [cols="2,3", options="header", stripes=even]
 |===
 |Label | Meaning
-|`db_operation_name` | Operation type (`add`, `delete`, `query`)
+|`db_operation_name` | Operation type (`add`, `upsert`, `delete`, `query`)
 |`db_system` | Vector DB/provider (`redis`, `chroma`, `pgvector`, …)
 |`spring_ai_kind` | `vector_store`
 |===