Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Refine readme #2

Merged
merged 1 commit into from
Apr 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,28 +4,24 @@
<a href="https://discord.gg/KqswhpVgdU"><img alt="discord invitation link" src="https://dcbadge.vercel.app/api/server/KqswhpVgdU?style=flat"></a>
<a href="https://twitter.com/TensorChord"><img src="https://img.shields.io/twitter/follow/tensorchord?style=social" alt="trackgit-views" /></a>

End-to-end service to query the text with hybrid search and rerank.
QText is a microservices framework for building the RAG pipeline, or semantic search engine on top of Postgres. It provides a simple API to add, query, and highlight the text in your existing database.

Application scenarios:
- Personal knowledge database + search engine
- Rerank experiment and visualization
- RAG pipeline
The main features include:

- Full-text search with Postgres GIN index.
- Vector and sparse search with [pgvecto.rs](https://github.com/tensorchord/pgvecto.rs)
- Reranking with cross-encoder model, cohere reranking API, or other methods.
- Semantic highlight

Besides this, qtext also provides a dashboard to visualize the vector search, sparse vector search, full text search, and reranking results.

[![asciicast](https://asciinema.org/a/653540.svg)](https://asciinema.org/a/653540)

## Features
## Design goals

- [x] full text search (Postgres GIN + text search)
- [x] vector similarity search ([pgvecto.rs](https://github.com/tensorchord/pgvecto.rs) HNSW)
- [x] sparse search ([pgvecto.rs](https://github.com/tensorchord/pgvecto.rs) HNSW)
- [x] generate vector and sparse vector if not provided
- [x] reranking
- [x] semantic highlight
- [x] hybrid search explanation
- [x] TUI
- [x] OpenAPI
- [x] OpenMetrics
- [ ] filtering
- **Simple**: easy to deploy and use.
- **Customizable**: can be integrated into your existing databases.
- **Extensible**: can be extended with new features.

## How to use

Expand All @@ -41,6 +37,10 @@ Some of the dependent services can be opt-out:
- `highlight`: used to provide the semantic highlight feature
- `encoder`: rerank with cross-encoder model, you can choose other methods or other online services

<div align="center">
<img src="./docs/images/arch.svg" alt="arch" width="500px">
</div>

For the client example, check:
- [test.py](./test.py): simple demo.
- [test_cohere_wiki.py](./test_cohere_wiki.py): a Wikipedia dataset with Cohere embedding.
Expand Down
3 changes: 3 additions & 0 deletions docs/images/arch.svg
kemingy marked this conversation as resolved.
Show resolved Hide resolved
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.