Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document paragraphs #1119

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Document paragraphs #1119

wants to merge 3 commits into from

Conversation

Horusiath
Copy link
Collaborator

@Horusiath Horusiath commented Jan 3, 2025

TODOs:

  • 1. Split documents by paragraphs, not arbitrary length (see: chore: replace document plain text with paragraphs AppFlowy-Collab#361)
  • 2. Create af_collab_embeddings.fragment_id using a consistent hash of chunk it relates to.
  • 3. At the beginning of indexing, get collab content with list of its currently indexed fragment_ids.
  • 4. When generating embeddings skip embeddings for those chunks which fragment_id was found in the point above and keep their content unchanged.
  • 5. When persisting embeddings don't delete all of the existing ones. Delete only those that were not found in point 3, those for which we didn't got embeddings stay the same, and insert all of the new ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant