feat: integrate with pgvector

[jiashenC]

I will need some feedback on the design here. @xzdandy @gaurav274

This PR reflects my initial design for in-database index features like pgvector. My initial idea is to offload as much as possible to the native database. When we create an index and do the index scan, we simply push an emulated query to the underlying database. While this is doable, this will introduce separate implementation paths in different components including the optimizer and the executor.

Other than the current design, another option is to reuse the third-party vector integration interface. There are also a few details that I am not clear about

• For other vector libraries, we keep an index entry in our own catalog. Do we still maintain that even for pgvector? In the original implementation, the index catalog entry is linked to a table catalog entry. Because in this case, the table is inside Postgres, we also need to change the implementation here a little bit.
• (Issue related to performance). Following the current create index implementation, data will be fetched from Postgres first but it is not really needed. The create index will anyway run inside Postgres.
• The current vector index scan is implemented based on _row_id. When it is scanning data from Postgres, we need to figure out a way to populate _row_id for the native database engine.

[xzdandy]

If the native database supports vector indexing, I think we should push down it to the native database system. This can be an optimizer rule. If the native database does not support vector indexing, we will do it ourselves.

[gaurav274]

Why are we not pushing the CREATE INDEX query to Postgres?

[gaurav274]

Based on offline discussion closing this PR

[jiashenC]

I reopen this PR because the index scan pass is also implemented in this branch.

This PR adds a feature to allow users to create an index on a table using pgvector and also do a similarity search using the existing pgvector index.

Create index query.

CREATE INDEX test_index ON test_data_source.test_vector (embedding) 
    USING PGVECTOR

When users attempt to create an index in pgvector, it is internally translated to a native Postgres query to push down the create index query.

Index scan.

SELECT idx, embedding FROM test_data_source.test_vector 
    ORDER BY Similarity(DummyFeatureExtractor(Open(...)), embedding)
    LIMIT 1

I take some shortcuts in the optimizer that if the data source is from Postgres, translate the similarity query to the semantically equivalent similarity query that works for Postgres and push down the query to Postgres. That is implemented as part of the VectorIndexScanExecutor.

[jarulraj]

What does other libraries mean?

[jiashenC]

I meant other vector store types that we currently support (e.g., FAISS).

[gaurav274]

Reminder to move it outside

[jiashenC]

@gaurav274 Added some changes. Can you give it a read? Not sure shall we still stick with the singledispatchmethod approach or do subclass inheritance and add some if statement like the Executor.

[xzdandy]

Overall design looks good to me. We may consider the gain/loss of moving some of the push down to optimizer. But I don't think that is the priority now. Left some comments for clarification.