Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLDR-455 change README and fixed 3.1, 3.3.4, 3.3.3 notes of FOND #323

Merged
merged 5 commits into from
Sep 5, 2023
Merged

TLDR-455 change README and fixed 3.1, 3.3.4, 3.3.3 notes of FOND #323

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b983d90
TLDR-455 change README and fixed 3.1, 3.3.4, 3.3.3 notes of FOND
oksidgy Sep 4, 2023
bcb8bca
TLDR-455 update installation instructions
oksidgy Sep 5, 2023
832ca00
TLDR-455 update README according online-instruction
oksidgy Sep 5, 2023
8819eff
TLDR-455 added info about trusted torch
oksidgy Sep 5, 2023
d4b80f7
fix indentation in code
dronperminov Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 150 additions & 22 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ It extracts a document’s logical structure and content, its tables, text forma
The document’s content is represented as a tree storing headings and lists of any level.
Dedoc can be integrated in a document contents and structure analysis system as a separate module.

Relevant documentation of the dedoc is available [here](https://dedoc.readthedocs.io).

## Features and advantages
Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats.
Document structure extraction is fully automatic regardless of input data type.
Expand All @@ -32,51 +30,181 @@ In 2022, the system won a grant to support the development of promising AI proje
* Using Tesseract, an actively developed OCR engine from Google, together with image preprocessing methods.
* Utilizing modern machine learning approaches for detecting a document orientation, detecting single/multicolumn document page, detecting bold text and extracting hierarchical structure based on the classification of features extracted from document images.


## Impact
This project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part).
Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems.
The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents.

# Online-Documentation
Relevant documentation of the dedoc is available [here]((https://dedoc.readthedocs.io/en/latest/))

# Installation instructions
****************************************
This project has REST Api and you can run it in Docker container.
Also, dedoc can be installed as a library via `pip`.
To read full Dedoc documentation go [here](https://dedoc.readthedocs.io).
There are two ways to install and run dedoc as a web application or a library that are described below.


## Run the project
## Install and run dedoc using docker

### Install and run dedoc using docker
You should have [`git`] (https://git-scm.com) and [`docker`](https://www.docker.com) installed for running dedoc by this method.
This method is more flexible because it doesn't depend on the operating system and other user's limitations,
still, the docker application should be installed and configured properly.

Clone the project
If you don't need to change the application configuration, you may use the built docker image as well.

### 1. Pull the image
```bash
docker pull dedocproject/dedoc
```

### 2. Run the container
```bash
docker run -p 1231:1231 --rm dedocproject/dedoc python3 /dedoc_root/dedoc/main.py
```

Go to [dockerhub](https://hub.docker.com/r/dedocproject/dedoc) to get more information about available dedoc images.

If you need to change some application settings, you may update `config.py` according to your needs and re-build the image.
You can build and run image:

### 1. Clone the repository
```bash
git clone https://github.com/ispras/dedoc
```

### 2. Go to the `dedoc` directory
```bash
git clone https://github.com/ispras/dedoc.git
cd dedoc
```

Ensure you have Docker installed.
Start `dedoc` on the port `1231`:
```bash

### 3. Build the image and run the application
```bash
docker-compose up --build
```

Start Dedoc with tests:
```bash
### 4. Run container with tests
```bash
test="true" docker-compose up --build
```
```

Now you can go to the `localhost:1231` and look at the docs and examples.
You can change the port and host in the config file `dedoc/config.py`.
If you need to change some application settings, you may update `config.py` according to your needs and re-build the image.

### Install dedoc using pip

One may install the dedoc library via `pip`.
To fulfil all the library requirements, you should have `torch~=1.11.0` and `torchvision~=0.12.0` installed.
You can install suitable for you versions of these libraries and install dedoc using `pip` command:
## Install dedoc using pip

If you don't want to use docker for running the application, it's possible to run dedoc locally.
However, it isn't suitable for any operating system (`Ubuntu 20+` is recommended) and
there may be not enough machine's resources for its work.
You should have `python` (`python3.8`, `python3.9` are recommended) and `pip` installed.

### 1. Install necessary packages:
```bash
sudo apt-get install -y libreoffice djvulibre-bin unzip unrar
```

`libreoffice` and `djvulibre-bin` packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf).
If you don't need converters, you can skip this step.
`unzip` and `unrar` packages are used in the process of extracting archives.

### 2. Install `Tesseract OCR 5` framework:
You can try any tutorial for this purpose or look [`here`](https://github.com/ispras/dedockerfiles/blob/master/dedoc_p3.9_base.Dockerfile)
to get the example of Tesseract installing for dedoc container or use next commands for building Tesseract OCR 5 from sources:

#### 2.1. Install compilers and libraries required by the Tesseract OCR:
```bash
sudo apt-get update
sudo apt-get install -y automake binutils-dev build-essential ca-certificates clang g++ g++-multilib gcc-multilib libcairo2 libffi-dev \
libgdk-pixbuf2.0-0 libglib2.0-dev libjpeg-dev libleptonica-dev libpango-1.0-0 libpango1.0-dev libpangocairo-1.0-0 libpng-dev libsm6 \
libtesseract-dev libtool libxext6 make pkg-config poppler-utils pstotext shared-mime-info software-properties-common swig zlib1g-dev
```
#### 2.2. Build Tesseract from sources:
```bash
sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
sudo apt-get update --allow-releaseinfo-change
sudo apt-get install -y tesseract-ocr tesseract-ocr-rus
git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/
cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd ..
export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/
```

## Install the dedoc library via pip.

To fulfil all the library requirements, you should have `torch~=1.11.0` and `torchvision~=0.12.0` installed.
You can install suitable for you versions of these libraries and install dedoc using pip command:

```bash
pip install dedoc
```

Or you can install dedoc with torch and torchvision included:

```bash
pip install "dedoc[torch]"
```

## Install and run dedoc from sources

If you want to run dedoc as a service from sources. it's possible to run dedoc locally.
However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and
there may be not enough machine's resources for its work.
You should have `python` (python3.8, python3.9 are recommended) and `pip` installed.

### 1. Install necessary packages: according to instructions [install necessary packages](#1-Install-necessary-packages)

### 2. Build Tesseract from sources according to instructions [Install Tesseract OCR-5 framework](#2-Install-Tesseract-OCR-5-framework)

### 3. We recommend to install python's virtual environment (for example, via `virtualenvwrapper`)

Below are the instructions for installing the package `virtualenvwrapper`:

```bash
sudo pip3 install virtualenv virtualenvwrapper
mkdir ~/.virtualenvs
export WORKON_HOME=~/.virtualenvs
echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8" >> ~/.bashrc
echo ". /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
source ~/.bashrc
mkvirtualenv dedoc_env
```

### 4. Install python's requirements and launch dedoc service on default port `1231`:

```bash
# clone dedoc project
git clone https://github.com/ispras/dedoc.git
cd dedoc
# check on your's python environment
workon dedoc_env
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt
pip install torch=1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
python dedoc/main.py -c ./dedoc/config.py
```
Now you can go to the `localhost:1231` and look at the docs and examples.

## Option: You can change the port of service:
you need to change environment DOCREADER_PORT

1. For local service launching on your_port (1166 example). [Install instruction from sources](#Install-and-run-dedoc-from-sources) and launch with environment:
```bash
DOCREADER_PORT=1166 python dedoc/main.py -c ./dedoc/config.py
```

2. For service launching in docker-container you need to change port value in DOCREADER_PORT env and field 'ports' in docker-compose.yml file:
```yaml
...
dedoc:
...
ports:
- your_port_number:your_port_number
environment:
DOCREADER_PORT: your_port_number
...
test:
...
environment:
DOCREADER_PORT: your_port_number
```

Go [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html) to get more details about dedoc installation.
86 changes: 76 additions & 10 deletions docs/source/getting_started/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,20 +59,25 @@ Install dedoc using pip
If you don't want to use docker for running the application, it's possible to run dedoc locally.
However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and
there may be not enough machine's resources for its work.
You should have `python` (python3.8+ is recommended) and `pip` installed.
You should have `python` (python3.8, python3.9 are recommended) and `pip` installed.

.. _install_packages:

1. Install `libreoffice` and `djvulibre-bin` packages:
1. Install necessary packages:
******************************

.. code-block:: bash

sudo apt-get install -y libreoffice djvulibre-bin
sudo apt-get install -y libreoffice djvulibre-bin unzip unrar

These packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf).
`libreoffice` and `djvulibre-bin` packages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf).
If you don't need converters, you can skip this step.
`unzip` and `unrar` packages are used in the process of extracting archives.

.. _install_tesseract:

2. Install `Tesseract OCR 5` framework.
***************************************
You can try any tutorial for this purpose or look `here <https://github.com/ispras/dedockerfiles/blob/master/dedoc_p3.9_base.Dockerfile>`_
to get the example of Tesseract installing for dedoc container or use next commands for building Tesseract OCR 5 from sources:

Expand All @@ -89,14 +94,17 @@ to get the example of Tesseract installing for dedoc container or use next comma

.. code-block:: bash

sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
sudo apt-get update --allow-releaseinfo-change
sudo apt-get install -y tesseract-ocr tesseract-ocr-rus
git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/
cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd ..
export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/
sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
sudo apt-get update --allow-releaseinfo-change
sudo apt-get install -y tesseract-ocr tesseract-ocr-rus
git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/
cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd ..
export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/

.. _install_library_via_pip:

3. Install the dedoc library via pip.
*************************************
To fulfil all the library requirements, you should have `torch~=1.11.0` and `torchvision~=0.12.0` installed.
You can install suitable for you versions of these libraries and install dedoc using pip command:

Expand All @@ -109,3 +117,61 @@ Or you can install dedoc with torch and torchvision included:
.. code-block:: bash

pip install "dedoc[torch]"

Install and run dedoc from sources
----------------------------------

If you want to run dedoc as a service from sources. it's possible to run dedoc locally.
However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and
there may be not enough machine's resources for its work.
You should have `python` (python3.8, python3.9 are recommended) and `pip` installed.

1. Install necessary packages: according to instructions :ref:`install_packages`

2. Build Tesseract from sources according to instructions :ref:`install_tesseract`

3. We recommend to install python's virtual environment (for example, via `virtualenvwrapper`)

Below are the instructions for installing the package `virtualenvwrapper`:

.. code-block:: bash

sudo pip3 install virtualenv virtualenvwrapper
mkdir ~/.virtualenvs
export WORKON_HOME=~/.virtualenvs
echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8" >> ~/.bashrc
echo ". /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
source ~/.bashrc
mkvirtualenv dedoc_env

4. Install python's requirements and launch dedoc service on default port `1231`:

.. code-block:: bash

# clone dedoc project
git clone https://github.com/ispras/dedoc.git
cd dedoc
# check on your's python environment
workon dedoc_env
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt
pip install torch=1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
python dedoc/main.py -c ./dedoc/config.py


Install trusted torch (verified version)
----------------------------------------------

You can install a trusted library `torch` (as a verified version of the library, verified by tools developed by the Ivannikov Institute for System Programming of the Russian Academy of Sciences).

For `python3.9`:
.. code-block:: bash

pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp39-cp39-linux_x86_64.whl
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp39-cp39-linux_x86_64.whl

For `python3.8`:
.. code-block:: bash

pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp38-cp38-linux_x86_64.whl
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp38-cp38-linux_x86_64.whl
Loading