diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d4bb2cb --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/api.rst b/docs/api.rst new file mode 100644 index 0000000..32146bd --- /dev/null +++ b/docs/api.rst @@ -0,0 +1,32 @@ +API Reference +============= + +.. automodule:: nightingale.cli + :members: + :undoc-members: + :show-inheritance: + +.. automodule:: nightingale.loader + :members: + :undoc-members: + :show-inheritance: + +.. automodule:: nightingale.mapper + :members: + :undoc-members: + :show-inheritance: + +.. automodule:: nightingale.config + :members: + :undoc-members: + :show-inheritance: + +.. automodule:: nightingale.utils + :members: + :undoc-members: + :show-inheritance: + +.. automodule:: nightingale.mapping.v1.config + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/cli.rst b/docs/cli.rst new file mode 100644 index 0000000..9b9c121 --- /dev/null +++ b/docs/cli.rst @@ -0,0 +1,20 @@ +CLI Reference +============= + +The following options are available for the Nightingale CLI tool: + +.. code-block:: sh + + nightingale --help + +Options +------- + +--config + Path to the configuration file. This option is required. + +--package + Package the data. This option is optional. + +--loglevel + Set the logging level. Available levels are: DEBUG, INFO, WARNING, ERROR, CRITICAL. diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..439e984 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,37 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information +import os +import sys + +sys.path.insert(0, os.path.abspath("../..")) + + +project = "nightingale" +copyright = "2024, Open Contracting" +author = "Open Contracting" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [ + "sphinx.ext.autodoc", + "sphinx.ext.napoleon", + "sphinx.ext.viewcode", + "sphinx.ext.todo", +] + + +templates_path = ["_templates"] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "alabaster" +html_static_path = ["_static"] diff --git a/docs/data_flow.rst b/docs/data_flow.rst new file mode 100644 index 0000000..e9fb683 --- /dev/null +++ b/docs/data_flow.rst @@ -0,0 +1,10 @@ +Data Flow +========= + +Nightingale transforms data through the following steps: + +1. Load flat data from the SQLite database using the `selector` from the configuration. +2. Read the mapping configuration to understand how to transform the data. +3. Map the data into the OCDS format. +4. Package the data if specified. +5. Write the data to the specified output. diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..9dad14d --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,25 @@ +.. nightingale documentation master file, created by + sphinx-quickstart on Fri Jun 14 13:51:42 2024. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to nightingale's documentation! +======================================= + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + installation + tutorial + data_flow + api + cli + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/installation.rst b/docs/installation.rst new file mode 100644 index 0000000..a932d03 --- /dev/null +++ b/docs/installation.rst @@ -0,0 +1,36 @@ +Installation +============ + +You can install the package using pip: + + .. code-block:: sh + + pip install nightingale + +Or you can install it from source: + +1. Clone the repository: + + .. code-block:: sh + + git clone https://github.com/open-contracting/nightingale + cd nightingale + +2. Create and activate a virtual environment: + + .. code-block:: bash + + python3 -m venv .venv + source .venv/bin/activate + +3. Install dependencies: + + .. code-block:: bash + + pip install -r requirements.txt + +4. Install the package: + + .. code-block:: bash + + pip install . diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..32bb245 --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/tutorial.rst b/docs/tutorial.rst new file mode 100644 index 0000000..642c1f2 --- /dev/null +++ b/docs/tutorial.rst @@ -0,0 +1,145 @@ +Tutorial +======== + +This tutorial will guide you through the basic usage of Nightingale, transforming flat data from an SQLite database into the OCDS format. + +Setup +----- + +1. **Prepare a Sample SQLite Database:** + + Create a SQLite database and populate it with some sample data. Here’s an example script (`create_sample_data.py`) to create a sample database: + + .. code-block:: python + + import sqlite3 + + # Connect to the database (or create it if it doesn't exist) + conn = sqlite3.connect('sample_database.db') + cursor = conn.cursor() + + # Create tables + cursor.execute(''' + CREATE TABLE example_table ( + id INTEGER PRIMARY KEY, + name TEXT, + value TEXT + ) + ''') + + cursor.execute(''' + CREATE TABLE party_table ( + id INTEGER PRIMARY KEY, + name TEXT, + identifier TEXT, + role TEXT + ) + ''') + + # Insert sample data into example_table + cursor.executemany(''' + INSERT INTO example_table (name, value) VALUES (?, ?) + ''', [ + ('sample1', 'value1'), + ('sample2', 'value2'), + ('sample3', 'value3'), + ]) + + # Insert sample data into party_table + cursor.executemany(''' + INSERT INTO party_table (name, identifier, role) VALUES (?, ?, ?) + ''', [ + ('party1', 'id1', 'buyer'), + ('party2', 'id2', 'supplier'), + ('party3', 'id3', 'procuringEntity'), + ]) + + # Commit and close + conn.commit() + conn.close() + + Run the script to create `sample_database.db`. + +2. **Create a Sample Configuration File:** + + Create a `sample_config.toml` file with the following content: + + .. code-block:: toml + + [datasource] + connection = 'sample_database.db' + + [mapping] + file = 'mapping.xlsx' + ocid_prefix = 'ocds-123abc' + force_publish = true + selector = ''' + SELECT + example_table.id AS "example_table (id)", + example_table.name AS "example_table (name)", + example_table.value AS "example_table (value)", + party_table.name AS "party_table (name)", + party_table.identifier AS "party_table (identifier)", + party_table.role AS "party_table (role)" + FROM example_table + JOIN party_table ON example_table.id = party_table.id; + ''' + + [publishing] + publisher = 'Sample Publisher' + version = '1.1' + + [output] + directory = 'output' + +3. **Prepare the Mapping File:** + + Use the following configuration for `mapping.xlsx` based on the `ocds field level mapping` template: + + **General Sheet:** + + .. code-block:: text + + | Title | Description | Path | Status | Mapping | Comment + |---------------|---------------|------------------|----------|------------------------------------ | ------- + | OCID | unique ID | ocid | Required | example_table (id) | - + | Party ID | Party ID | parties/id | Optional | party_table (identifier) | - + | Party Name | Party Name | parties/name | Optional | party_table (name) + | Role | Role | parties/roles | Required | party_table (role) | - + + + **Tender Sheet:** + + .. code-block:: text + + id | Title | Description | Path | Status | Mapping | Comment + ----|----------------|--------------|-------------------|----------|------------------------------------ | ------- + 1 | Tender Title | Tender title | tender/title | Optional | example_table (name) | - + 2 | Value | Tender value | tender/value/amount | Optional | example_table (value) | - + + +Running the Transformation +-------------------------- + +Run the transformation using the CLI: + +.. code-block:: sh + + nightingale --config sample_config.toml --package --loglevel DEBUG + +This will produce an output file in the `output` directory. + +Mapping Configuration +---------------------- + +Field-level mapping is specified in the `mapping.xlsx` file. It is formed from stardard `"OCDS Field Level Mapping template" `_. +For more information about how to fill the mapping file, refer to the `OCDS Field Level Mapping template guidance `_. + +Here the bried description of the columns from mapping sheets in the mapping file: + + * **Path**: The path in the OCDS release schema where the field value should be placed. + * **Title**: A human-readable title for the field. + * **Description**: A description of what the field represents. + * **Mapping**: The field in the source data that maps to the OCDS path. + +Understanding these mappings will help you configure the transformation correctly for your data.