Release 0.1.0-20240906 (#203)

* Replace PaiEas LLM with LLI-integration and upgrade python to 3.11 (#148)

* Replace PaiEas LLM with LLI-integration and upgrade python version to 3.11

* Replace MyFCDashScope with OpenAILike class

* Fix pyproject dependency

* bug fix (#149)

* Support postgresql load user dict (#150)

* make format

* Allow not install extension pg_jieba

* table name data_default

* Convert raptor processor to TransformComponent (#151)

* udpate raptor using transform

* modify raptor with transform

* modify raptor and dataloader

---------

Co-authored-by: Yue Fei <[email protected]>

* Add clip model (#130)

* Update

* Add clip model

* Fix oss cache

* Fix cache

* Pdf reader upload image

* Add multimodal

* Update config

* Use two embedding

* Add text_image node

* Add tests

* Fix tests

* fix multi_modal_vector

---------

Co-authored-by: 燃夏 <[email protected]>

* Fix docker base image (#152)

* change insert to be sync (#153)

* Personal/ranxia/fix image readme (#155)

* fix multi_modal and readme

* fix multi_modal and readme

* fix multi_modal and readme

* fix multi_modal image (#156)

* Support Agentic RAG with intent and functioncalling (#154)

* Add intent detection module

* Remove LlmQuery class

* Support API

* Refactor agent module and format toml

* Refactor module tool

* Refactor query api

* Add demo and UI

* remove

* Fix reviews

* Add test for intent and api

* Add web search (#161)

* Add web search

* Fix lint

* Fix bug

* Update timeout

* Fix bug

* Fix jieba bug (#163)

* Support PAI-EAS MultiModal LLM (#168)

* Support minicpm

* Fix issue

* Bugfix: PaiEas LLM endpoint & max_tokens (#171)

* Fix dashscope interface (#172)

* Fix dashscope llm

* Fix bug

* Fix test bug (#174)

* add minerU (#160)

* add minerU

* add minerU

* add minerU

* Fix nodes id and simi_topK

* remove image url from text

* remove image url from text

* remove image url from text

* Support FAQ query w/o image (#162)

* Support FAQ query w/o image

* Using LLM when query w/o images

* Personal/ranxia/mineru enhancement (#164)

* remove repeat nodes

* show multiple pictures in media

* show multiple pictures in media

* Install miner with poetry (#165)

* fix retriever

* Support OSS Data Loader (#166)

* Support oss data loader

* Skip file which has been uploaded

* Support oss prefix via api

* 1. change image size (#167)

2. limit image number
3. fix retriever answer ui format

* adjust image score (#169)

* merge feature

* merge feature

* merge feature

* merge feature

* Fix bug (#173)

* Support chunk text-overflow display (#170)

* Fix bugs

* Support text-overflow

* Support text-overflow

* Support load MinerU config file automatically (#175)

* Support load MinerU config file automatically

* Modify

* Direct writing the config rather than copying

* Fix multi_modal build docker (#176)

* fix load_config (#177)

* change  multimodal prompt (#178)

* Test Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix test bug (#174) (#179)

Co-authored-by: Yue Fei <[email protected]>

* Fix Dockerfile (#180)

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix Dockerfile

* Fix docker env (#181)

* Fix Dockerfile

* Fix bugs

* Fix docker env

* Fix docker env

* Fix docker env (#183)

* Fix Dockerfile

* Fix bugs

* Fix docker env

* Fix docker env

* Fix docker env

* Fix docker env

* Fix docker env

* Bugfix

* Bugfix for EAS (#184)

* Fix Dockerfile

* Fix bugs

* Fix docker env

* Fix docker env

* Fix docker env

* Fix docker env

* Fix docker env

* Bugfix

* Bugfix

* Fix detectron link (#182)

* Update detectron dependency (#185)

* Update dependency

* udpate poetry lock

* fix multimodal_config and prompt (#186)

* fix MinerU readme (#189)

* Add timeout and more logs (#188)

* Personal/ranxia/fix miner u readme (#190)

* fix MinerU readme

* fix MinerU readme

* Personal/ranxia/fix miner u readme (#191)

* fix MinerU readme

* fix MinerU readme

* fix MinerU config

* fix MinerU bug (#192)

* Personal/ranxia/fix test and review bug (#193)

* fix MinerU bug

* fix MinerU bug

* fix MinerU bug

* fix MinerU bug

* fix MinerU bug

* fix MinerU bug

* fix MinerU bug

---------

Co-authored-by: 筱文 <[email protected]>
Co-authored-by: Yue Fei <[email protected]>

* fix multimodal readme and config (#195)

* nl2sql refactoring (#194)

* change insert to be sync

* add nl2sql

* nl2sql setting

* nl2sql setting

* fix test bug

* fix bugs

* data analysis retriever and synthesizer

* fix tests bugs

* add data_analysis ui

* update poetry.lock

* remove unnecessary comment

* add fault tolerance if no file provided

* add minor fault tolerance

* add upload_datasheet

* nl2sql refactor and add db ui

* restore retriever & synthesizer

* update poetry.lock

* Fix list merge

* bug fix

* add default display

---------

Co-authored-by: 陆逊 <[email protected]>

* Personal/xi/nl2sql UI (#196)

* change insert to be sync

* add nl2sql

* nl2sql setting

* nl2sql setting

* fix test bug

* fix bugs

* data analysis retriever and synthesizer

* fix tests bugs

* add data_analysis ui

* update poetry.lock

* remove unnecessary comment

* add fault tolerance if no file provided

* add minor fault tolerance

* add upload_datasheet

* nl2sql refactor and add db ui

* restore retriever & synthesizer

* update poetry.lock

* Fix list merge

* bug fix

* add default display

* data_analysis ui update

---------

Co-authored-by: 陆逊 <[email protected]>

* Personal/ranxia/change max new tokens (#199)

* set multimodal llm max_new_tokens

* set multimodal llm max_new_tokens

* Add trace (#197)

* Add trace

* Fix bug

* Push to hangzhou region by default

* 修复tables和descriptions默认配置bug (#198)

* change insert to be sync

* add nl2sql

* nl2sql setting

* nl2sql setting

* fix test bug

* fix bugs

* data analysis retriever and synthesizer

* fix tests bugs

* add data_analysis ui

* update poetry.lock

* remove unnecessary comment

* add fault tolerance if no file provided

* add minor fault tolerance

* add upload_datasheet

* nl2sql refactor and add db ui

* restore retriever & synthesizer

* update poetry.lock

* Fix list merge

* bug fix

* add default display

* data_analysis ui update

* fix table & description & query_output bugs

* fix inconsistency between frontend and backend data structures

---------

Co-authored-by: 陆逊 <[email protected]>

* Fix nginx routing (#200)

* Fix nginx routing (#202)

* Fix nginx routing

* Fix nginx config

* add data_analysis doc (#201)

Co-authored-by: Yue Fei <[email protected]>

* Resolve conflict

---------

Co-authored-by: wwxxzz <[email protected]>
Co-authored-by: aero-xi <[email protected]>
Co-authored-by: zt2645802240 <[email protected]>
Co-authored-by: 燃夏 <[email protected]>

.github/workflows/ci.yml

@@ -19,12 +19,12 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
-      - name: Set up Python 3.10
+      - name: Set up Python 3.11
         # This is the version of the action for setting up Python, not the Python version.
         uses: actions/setup-python@v5
         with:
           # Semantic version range syntax or exact version of a Python version
-          python-version: "3.10"
+          python-version: "3.11"
           # Optional - x64 or x86 architecture, defaults to x64
           architecture: "x64"
 

.github/workflows/docker.yml

@@ -8,7 +8,7 @@ on:
 
 # Defines two custom environment variables for the workflow. These are used for the Container registry domain, and a name for the Docker image that this workflow builds.
 env:
-  REGISTRY: registry.cn-beijing.aliyuncs.com
+  REGISTRY: registry.cn-hangzhou.aliyuncs.com
 
 # There is a single job in this workflow. It's configured to run on the latest available version of Ubuntu.
 jobs:
@@ -36,7 +36,7 @@ jobs:
           echo "VERSION_TAG=$VERSION_TAG" >> $GITHUB_ENV
 
       # Uses the `docker/login-action` action to log in to the Container registry registry using the account and password that will publish the packages. Once published, the packages are scoped to the account defined here.
-      - name: Login to ACR Beijing region
+      - name: Login to ACR Hangzhou region
         uses: docker/login-action@v1
         with:
           registry: ${{ env.REGISTRY }}

.github/workflows/main.yml

@@ -19,12 +19,12 @@ jobs:
 
     steps:
       - uses: actions/checkout@v4
-      - name: Set up Python 3.10
+      - name: Set up Python 3.11
         # This is the version of the action for setting up Python, not the Python version.
         uses: actions/setup-python@v5
         with:
           # Semantic version range syntax or exact version of a Python version
-          python-version: "3.10"
+          python-version: "3.11"
           # Optional - x64 or x86 architecture, defaults to x64
           architecture: "x64"
 
@@ -59,6 +59,6 @@ jobs:
         with:
           coverageFile: localdata/test_output/coverage_report.xml
           token: ${{ secrets.GITHUB_TOKEN }}
-          thresholdAll: 0.5 # Total coverage threshold
+          thresholdAll: 0.4 # Total coverage threshold
           #thresholdNew: 0.9 # New files coverage threshold
           #thresholdModified: 0.9 # Modified files coverage threshold

.github/workflows/main_gpu.yml

@@ -18,13 +18,20 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
+      - name: Free disk space
+        run: |
+          sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
+          sudo rm -rf \
+            /usr/share/dotnet /usr/local/lib/android /opt/ghc \
+            /usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
+            /usr/lib/jvm || true
       - uses: actions/checkout@v4
-      - name: Set up Python 3.10
+      - name: Set up Python 3.11
         # This is the version of the action for setting up Python, not the Python version.
         uses: actions/setup-python@v5
         with:
           # Semantic version range syntax or exact version of a Python version
-          python-version: "3.10"
+          python-version: "3.11"
           # Optional - x64 or x86 architecture, defaults to x64
           architecture: "x64"
 

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.10-slim AS builder
+FROM python:3.11 AS builder
 
 RUN pip3 install poetry
 
@@ -12,7 +12,7 @@ COPY . .
 
 RUN poetry install && rm -rf $POETRY_CACHE_DIR
 
-FROM python:3.10-slim AS prod
+FROM python:3.11-slim AS prod
 
 RUN rm -rf /etc/localtime && ln -s /usr/share/zoneinfo/Asia/Harbin  /etc/localtime
 

Dockerfile_gpu

@@ -1,4 +1,4 @@
-FROM python:3.10-slim AS builder
+FROM python:3.11 AS builder
 
 RUN pip3 install poetry
 
@@ -14,7 +14,7 @@ RUN mv pyproject_gpu.toml pyproject.toml \
 
 RUN poetry install && rm -rf $POETRY_CACHE_DIR
 
-FROM python:3.10-slim AS prod
+FROM python:3.11-slim AS prod
 
 RUN rm -rf /etc/localtime && ln -s /usr/share/zoneinfo/Asia/Harbin  /etc/localtime
 

Dockerfile_ui

@@ -1,4 +1,4 @@
-FROM python:3.10-slim AS builder
+FROM python:3.11 AS builder
 
 RUN pip3 install poetry
 
@@ -12,7 +12,7 @@ COPY . .
 
 RUN poetry install && rm -rf $POETRY_CACHE_DIR
 
-FROM python:3.10-slim AS prod
+FROM python:3.11-slim AS prod
 
 RUN rm -rf /etc/localtime && ln -s /usr/share/zoneinfo/Asia/Harbin  /etc/localtime
 

README.md

@@ -48,31 +48,31 @@ PAI-RAG is an easy-to-use opensource framework for modular RAG (Retrieval-Augmen
 
 2. Development Environment Settings
 
-   This project uses poetry for management. To ensure environmental consistency and avoid problems caused by Python version differences, we specify Python version 3.10.
+   This project uses poetry for management. To ensure environmental consistency and avoid problems caused by Python version differences, we specify Python version 3.11.
 
    ```bash
-   conda create -n rag_env python==3.10
+   conda create -n rag_env python==3.11
    conda activate rag_env
    ```
 
-- (1) CPU
+### (1) CPU
 
-  Use poetry to install project dependency packages directly:
+Use poetry to install project dependency packages directly:
 
-  ```bash
-  pip install poetry
-  poetry install
-  ```
+```bash
+pip install poetry
+poetry install
+```
 
-- (2) GPU
+### (2) GPU
 
-  First replace the default pyproject.toml with the GPU version, and then use poetry to install the project dependency package:
+First replace the default pyproject.toml with the GPU version, and then use poetry to install the project dependency package:
 
-  ```bash
-  mv pyproject_gpu.toml pyproject.toml && rm poetry.lock
-  pip install poetry
-  poetry install
-  ```
+```bash
+mv pyproject_gpu.toml pyproject.toml && rm poetry.lock
+pip install poetry
+poetry install
+```
 
 - Common network timeout issues
 
@@ -119,13 +119,35 @@ PAI-RAG is an easy-to-use opensource framework for modular RAG (Retrieval-Augmen
    export DASHSCOPE_API_KEY=""
    ```
 
+   To utilize Object Storage Service (OSS) for file storage, particularly when operating in multimodal mode, you must first configure settings in both the src/pai_rag/config/settings.toml and src/pai_rag/config/settings_multi_modal.toml configuration files. Append the following TOML configuration snippet within these files:
+
+   ```toml
+   [rag.oss_store]
+   bucket = ""
+   endpoint = ""
+   prefix = ""
+   ```
+
+   Additionally, you need to introduce environment variables:
+
+   ```bash
+   export OSS_ACCESS_KEY_ID=""
+   export OSS_ACCESS_KEY_SECRET=""
+   ```
+
    ```bash
    # Support custom host (default 0.0.0.0), port (default 8001), config (default src/pai_rag/config/settings.yaml), enable-example (default True), skip-download-models (default False)
    # Download [bge-small-zh-v1.5, easyocr] by default, you can skip it by setting --skip-download-models.
    # you can use tool "load_model" to download other models including [bge-small-zh-v1.5, easyocr, SGPT-125M-weightedmean-nli-bitfit, bge-large-zh-v1.5, bge-m3, bge-reranker-base, bge-reranker-large, paraphrase-multilingual-MiniLM-L12-v2, qwen_1.8b, text2vec-large-chinese]
    pai_rag serve [--host HOST] [--port PORT] [--config CONFIG_FILE] [--enable-example False] [--skip-download-models]
    ```
 
+   The default configuration file is src/pai_rag/config/settings.yaml. However, if you require the multimodal llm module, you should switch to the src/pai_rag/config/settings_multi_modal.yaml file instead.
+
+   ```bash
+   pai_rag serve -c src/pai_rag/config/settings_multi_modal.yaml
+   ```
+
 5. Download provided models to local directory
 
    ```bash
@@ -325,10 +347,14 @@ curl -X 'POST' http://127.0.0.1:8000/service/evaluate/retrieval
 curl -X 'POST' http://127.0.0.1:8000/service/evaluate/response
 ```
 
-# Function Calling
+# Agentic RAG
+
+You can use agent with function calling api-tools in PAI-RAG, please refer to the documentation:
+[Agentic RAG](./example_data/function_tools/api-tool-with-intent-detection-for-travel-assistant/README.md)
+
+# Data Analysis
 
-You can use function calling tools in PAI-RAG, please refer to the documentation:
-[Function Calling Instruction](./docs/function_calling/readme.md)
+You can use data analysis based on database or sheet file in PAI-RAG, please refer to the documentation: [Data Analysis](./docs/data_analysis_doc.md)
 
 # Parameter Configuration
 

README_zh.md

@@ -43,31 +43,31 @@ PAI-RAG 是一个易于使用的模块化 RAG（检索增强生成）开源框
 
 2. 配置开发环境
 
-   本项目使用poetry进行管理，若在本地环境下使用，建议在安装环境之前先创建一个空环境。为了确保环境一致性并避免因Python版本差异造成的问题，我们指定Python版本为3.10。
+   本项目使用poetry进行管理，若在本地环境下使用，建议在安装环境之前先创建一个空环境。为了确保环境一致性并避免因Python版本差异造成的问题，我们指定Python版本为3.11。
 
    ```bash
-   conda create -n rag_env python==3.10
+   conda create -n rag_env python==3.11
    conda activate rag_env
    ```
 
-- (1) CPU环境
+   ### (1) CPU环境
 
-  直接使用poetry安装项目依赖包：
+   直接使用poetry安装项目依赖包：
 
-  ```bash
-  pip install poetry
-  poetry install
-  ```
+   ```bash
+    pip install poetry
+    poetry install
+   ```
 
-- (2) GPU环境
+### (2) GPU环境
 
-  首先替换默认 pyproject.toml 为 GPU 版本, 再使用poetry安装项目依赖包：
+首先替换默认 pyproject.toml 为 GPU 版本, 再使用poetry安装项目依赖包：
 
-  ```bash
-  mv pyproject_gpu.toml pyproject.toml && rm poetry.lock
-  pip install poetry
-  poetry install
-  ```
+```bash
+mv pyproject_gpu.toml pyproject.toml && rm poetry.lock
+pip install poetry
+poetry install
+```
 
 - 常见网络超时问题
 
@@ -119,13 +119,37 @@ PAI-RAG 是一个易于使用的模块化 RAG（检索增强生成）开源框
    export DASHSCOPE_API_KEY=""
    ```
 
+   使用OSS存储文件(使用多模态模式时必须提前配置)，在配置文件src/pai_rag/config/settings.toml和src/pai_rag/config/settings_multi_modal.toml中添加以下配置:
+
+   ```toml
+   [rag.oss_store]
+   bucket = ""
+   endpoint = ""
+   prefix = ""
+   ```
+
+   并需要在命令行引入环境变量
+
+   ```bash
+   export OSS_ACCESS_KEY_ID=""
+   export OSS_ACCESS_KEY_SECRET=""
+   ```
+
+   启动RAG服务
+
    ```bash
    # 启动，支持自定义host(默认0.0.0.0), port(默认8001), config(默认src/pai_rag/config/settings.yaml), enable-example(默认True), skip-download-models(不加为False)
    # 默认启动时下载模型 [bge-small-zh-v1.5, easyocr] , 可设置 skip-download-models 避免启动时下载模型.
    # 可使用命令行 "load_model" 下载模型 including [bge-small-zh-v1.5, easyocr, SGPT-125M-weightedmean-nli-bitfit, bge-large-zh-v1.5, bge-m3, bge-reranker-base, bge-reranker-large, paraphrase-multilingual-MiniLM-L12-v2, qwen_1.8b, text2vec-large-chinese]
    pai_rag serve [--host HOST] [--port PORT] [--config CONFIG_FILE] [--enable-example False] [--skip-download-models]
    ```
 
+   启动默认配置文件为src/pai_rag/config/settings.yaml，若需要使用多模态，请切换到src/pai_rag/config/settings_multi_modal.yaml
+
+   ```bash
+   pai_rag serve -c src/pai_rag/config/settings_multi_modal.yaml
+   ```
+
 5. 下载其他模型到本地
 
    ```bash
@@ -275,11 +299,14 @@ curl -X 'POST' http://127.0.0.1:8000/service/evaluate/retrieval
 curl -X 'POST' http://127.0.0.1:8000/service/evaluate/response
 ```
 
-# Function Calling
+# Agentic RAG
+
+您也可以在PAI-RAG中使用支持API function calling功能的Agent，请参考文档：
+[Agentic RAG](./example_data/function_tools/api-tool-with-intent-detection-for-travel-assistant/README.md)
 
-您也可以在PAI-RAG中使用function calling功能，请参考文档：
+# Data Analysis
 
-[Function Calling 使用说明](./docs/function_calling/readme.md)
+您可以在PAI-RAG中使用支持数据库和表格文件的数据分析功能，请参考文档：[Data Analysis](./docs/data_analysis_doc.md)
 
 # 参数配置
 

docs/data_analysis_doc.md

@@ -0,0 +1,64 @@
+# 模型配置
+
+在web界面 Settings 中，左侧下方选择需要的LLM，如果选择DashScope（通义API），推荐使用qwen-max模型；如果选择PaiEas开源部署，推荐使用qwen2-72b-instruct模型
+
+点击右侧button更新使用的模型
+
+![llm_selection](/docs/figures/data_analysis/llm_selection.png)
+
+点击web界面上方Data Analysis，进入到数据分析页面，支持两种类型的数据分析：连接数据库（mysql）分析 和 上传表格文件（excel/csv）分析
+
+![data_analysis_overview](/docs/figures/data_analysis/data_analysis_overview.png)
+
+# 数据库分析配置
+
+## 数据库连接
+
+连接数据库，选择左上方数据分析类型为 database，出现数据库连接配置界面，如下图：
+
+![db_config](/docs/figures/data_analysis/db_config.png)
+
+其中，
+
+- Dialect为数据库类别，当前支持mysql，默认mysql
+- Username和Passoword分别为用户名和密码
+- Host为本地或远程数据库url，Port为接口，默认3306
+- DBname为需要分析的目标数据库名称
+- Tables为需要分析的数据表，格式为：table_A, table_B,... ，默认为空，使用目标数据库中所有数据表
+- Descriptions为针对目标数据库中每张表的补充描述，比如对表中字段的进一步解释，可以提升数据分析效果，格式为：{"table_A":"字段a表示xxx，字段b数据的格式为yyy","table_B":"这张表主要用于zzz"}，注意：需要使用英文输入法下的字典格式（英文双引号，冒号，逗号），默认为空
+
+填好以上信息后，点击左侧下方Connect Database按钮，看到Connection info如下图，表示连接成功，可以在右侧chatbot中进行提问
+
+![db_connect](/docs/figures/data_analysis/db_connect.png)
+
+如果需要更新数据库，重新填写以上信息，点击Connect Dtabase即可
+
+## 查询效果优化
+
+针对数据表中字段含义不清晰，或者字段存储内容格式不清晰等问题，可以在Descriptions中增加相应描述，帮助llm更准确提取数据表内容，此处以公开数据集Spider中my_pets数据库为例，其中pets表数据如下：
+
+![table_example](/docs/figures/data_analysis/table_example.png)
+
+问答效果对比：
+
+当描述为空时，对问题“有几只狗”生成的sql查询语句为：SELECT COUNT(\*) FROM pets WHERE PetType = '狗'，查询不到
+
+![db_query_no_desc](/docs/figures/data_analysis/db_query_no_desc.png)
+
+增加简单描述后，生成的sql查询语句为：SELECT COUNT(\*) FROM pets WHERE PetType = 'Dog'，可以准确回答
+
+![db_query_desc](/docs/figures/data_analysis/db_query_desc.png)
+
+如果查询效果有明显改善，可以将相应的补充描述在数据库中作为相应table或column的comment持久化添加
+
+# 表格文件分析配置
+
+表格文件配置相对简单，选择左上方的分析类型为：datafile，出现以下界面
+
+![sheet_upload](/docs/figures/data_analysis/sheet_upload.png)
+
+点击左侧中部的上传，一次上传一份表格文件（excel或csv格式），上传成功后，左侧下方会出现文件的前几行预览，如下图所示：
+
+![sheet_data_preview](/docs/figures/data_analysis/sheet_data_preview.png)
+
+上传表格文件后可以直接在右侧chatbot中提问，如需更换表格，重新上传所需表格即可