Support similarity scores in Document API #1794
Conversation
| * @author Thomas Vitale | ||
| * @since 1.0.0 | ||
| */ | ||
| public enum DocumentMetadata { |
There was a problem hiding this comment.
The idea for this enum is to use it for other common metadata used in Documents, such as the "source file" or "page" when using a DocumentReader, helping the RAG flow traceability.
| * The lower the distance, the more they are similar. | ||
| * It's the opposite of the similarity score. | ||
| */ | ||
| DISTANCE("distance"); |
There was a problem hiding this comment.
I kept this metadata for backward compatibility, but we might consider removing it completely since we now have the "score" field in each Document (and "distance" is always the opposite value of "score").
| .filter(s -> s.score >= request.getSimilarityThreshold()) | ||
| .sorted(Comparator.<Similarity>comparingDouble(s -> s.score).reversed()) | ||
| .peek(document -> document | ||
| .setScore(EmbeddingMath.cosineSimilarity(userQueryEmbedding, document.getEmbedding()))) |
There was a problem hiding this comment.
If we remove the "embedding" field (see: #1781), the SimpleVectorStore will not work
| // It must always be "latest" or else Azure locks the image after a while. See: | ||
| // https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/60 | ||
| public static final DockerImageName DEFAULT_IMAGE = DockerImageName | ||
| .parse("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest"); |
There was a problem hiding this comment.
I hope we'll be able to have integration tests for CosmosDB based on Testcontainers in the future. For now, this image includes the vector store-specific features disabled and there's no way to enable them, so it cannot be used.
| @@ -298,6 +303,8 @@ public static final class PineconeVectorStoreConfig { | |||
|
|
|||
| private final String contentFieldName; | |||
|
|
|||
| // TODO: Why is this field configurable? Can we remove this after standardizing | |||
There was a problem hiding this comment.
Would it be ok to remove this and keep the standard "distance" metadata? Having this configurable means we cannot use the metadata reliably across implementations.
| Map<String, Object> metadata = this.config.metadataFields.stream() | ||
| .map(MetadataField::name) | ||
| .filter(doc::hasProperty) | ||
| .collect(Collectors.toMap(Function.identity(), doc::getString)); | ||
| // TODO: this seems wrong. The key is named "vector_store", but the value is the |
There was a problem hiding this comment.
Would it be ok to remove this and keep the standard "distance" metadata?
|
PR updated after #1822 was merged |
ThomasVitale
force-pushed
the
similarity-score-documents
branch
from
467dbc3
to
efca3c7
Compare
|
|
||
| public void setScore(@Nullable Double score) { | ||
| this.score = score; | ||
| } |
There was a problem hiding this comment.
Should the document object be immutable? If we want to update a Document with a score we should use a builder that takes the existing document and then call the 'score' builder method?
There was a problem hiding this comment.
The post-retrieval steps in a RAG flow would all modify the Documents in some way, including the score, the content and the metadata. I guess we could create new instances on each hop to the next step in the flow. But we'd need several builders based on the type of field we change. Should I go with that?
There was a problem hiding this comment.
I made score a final field and introduced a mutate() method to build a new Document instance with the possibility to change the score. However, Document was not immutable to begin with (media and metadata are mutable). I have created a separate issue to look into that because it would be a breaking change and it would require lots of refactoring. #1838
| /** | ||
| * @deprecated Use builder instead: {@link Document#builder()}. | ||
| */ | ||
| @Deprecated(since = "1.0.0-M5", forRemoval = true) | ||
| public Document(String content, Map<String, Object> metadata) { |
There was a problem hiding this comment.
There are probably many users out there using this constructor and perhaps the one with String id, String content, Map<String, Object> metadata args. Despite the having the builder, maybe we keep these ctors?
There was a problem hiding this comment.
I thought of deprecating most of the constructors to make the callers more readable, mostly a problem with the other varying constructors with 3 or more arguments. For example, there are 3 constructors that accept 3 arguments, but all different (with media and metadata very easy to mix-up).
What if keep only the 2 ones you mentioned? Or should I keep all of them?
There was a problem hiding this comment.
I have "undeprecated" the mentioned constructors.
Document * Introduced “score” attribute in Document API. It stores the similarity score. * Consolidate “distance” metadata for Documents. It stores the distance measurement. * Adopted prefix-less naming convention in Document.Builder and deprecated old methods. * Deprecated the many overloaded Document constructors in favour of Document.Builder. Vector Stores * Every vector store implementation now configures a “score” attribute with the similarity score of the Document embedding. It also includes the “distance” metadata with the distance measurement. * Fixed error in Elasticsearch where distance and similarity were mixed up. * Added missing integration tests for SimpleVectorStore. * The Azure Vector Store and HanaDB Vector Store do not include those measurements because the product documentation do not include information about how the similarity score is returned, and without access to the cloud products I could not verify that via debugging. * Improved tests to actually assert the result of the similarity search based on the returned score. Signed-off-by: Thomas Vitale <[email protected]>
Signed-off-by: Thomas Vitale <[email protected]>
ThomasVitale
force-pushed
the
similarity-score-documents
branch
from
4758e9e
to
3a740ec
Compare
Document
Vector Stores