Repo showcasing AI summarization tool.
This repo showcases a simple, yet effective tool for document summarization. It can work with plain-text and PDF documents in any language supported by underlying LLM (Mistral by default).
Install following dependencies (on macOS):
- Python 3 installation (e.g. Miniconda or Homebrew package)
- Python packages: run
pip3 install -r requirements.txt - Download
mistral-7b-openorca.Q5_K_M.ggufmodel from Hugging Face TheBloke/Mistral-7B-OpenOrca-GGUF repo into localmodelsdirectory.
Note you can experiment with anternatives models, just update the MODEL_FILE and MODEL_CONTEXT_WINDOW variables in web-ui.py and/or Notebook.ipynb.
In order to run Web UI just run python3 ./web-ui.py in the repo folder. This should open Web UI interface in the browser.
The tool can be used as Jupyter Labs/Notebook as well, you open the Notebook.ipynb in Jupyter Labs.
Depending on the document size, this tool works in following modes:
- In the simple case, if the whole document can fit into model's context window then summarizartion is based on adding relevant summarization prompt.
- In case of large documents, document processed using "map-reduce" pattern:
- The document is first split into smaller chunks using `RecursiveCharacterTextSplitter`` which tries to respect paragraph and sentence boundarions.
- Each chunk is summarized separately (map step).
- Chunk summarization are summarized again to give final summary (reduce step).
All processing is done locally on the user's machine.
- Quantified Mistral model (
mistral-7b-openorca.Q5_K_M.gguf) has around 5,1 GB.
Relatively small to medium documents (couple of pages) should fit into single context window, which results in processing time of around 40s on Apple MBP with M1 chip.
None know issue.