add docs

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/api.rst

@@ -0,0 +1,32 @@
+API Reference
+=============
+
+.. automodule:: nightingale.cli
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: nightingale.loader
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: nightingale.mapper
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: nightingale.config
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: nightingale.utils
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: nightingale.mapping.v1.config
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/cli.rst

@@ -0,0 +1,20 @@
+CLI Reference
+=============
+
+The following options are available for the Nightingale CLI tool:
+
+.. code-block:: sh
+
+    nightingale --help
+
+Options
+-------
+
+--config <path>
+    Path to the configuration file. This option is required.
+
+--package
+    Package the data. This option is optional.
+
+--loglevel <level>
+    Set the logging level. Available levels are: DEBUG, INFO, WARNING, ERROR, CRITICAL.

docs/conf.py

@@ -0,0 +1,37 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../.."))
+
+
+project = "nightingale"
+copyright = "2024, Open Contracting"
+author = "Open Contracting"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.todo",
+]
+
+
+templates_path = ["_templates"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "alabaster"
+html_static_path = ["_static"]

docs/data_flow.rst

@@ -0,0 +1,10 @@
+Data Flow
+=========
+
+Nightingale transforms data through the following steps:
+
+1. Load flat data from the SQLite database using the `selector` from the configuration.
+2. Read the mapping configuration to understand how to transform the data.
+3. Map the data into the OCDS format.
+4. Package the data if specified.
+5. Write the data to the specified output.

docs/index.rst

@@ -0,0 +1,25 @@
+.. nightingale documentation master file, created by
+   sphinx-quickstart on Fri Jun 14 13:51:42 2024.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to nightingale's documentation!
+=======================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   installation
+   tutorial
+   data_flow
+   api
+   cli
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/installation.rst

@@ -0,0 +1,36 @@
+Installation
+============
+
+You can install the package using pip:
+
+    .. code-block:: sh
+
+        pip install nightingale
+
+Or you can install it from source:
+
+1. Clone the repository:
+
+    .. code-block:: sh
+
+        git clone https://github.com/open-contracting/nightingale
+        cd nightingale
+
+2. Create and activate a virtual environment:
+
+    .. code-block:: bash
+
+        python3 -m venv .venv
+        source .venv/bin/activate
+
+3. Install dependencies:
+
+    .. code-block:: bash
+
+        pip install -r requirements.txt
+
+4. Install the package:
+
+    .. code-block:: bash
+
+        pip install .

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/tutorial.rst

@@ -0,0 +1,145 @@
+Tutorial
+========
+
+This tutorial will guide you through the basic usage of Nightingale, transforming flat data from an SQLite database into the OCDS format.
+
+Setup
+-----
+
+1. **Prepare a Sample SQLite Database:**
+
+   Create a SQLite database and populate it with some sample data. Here’s an example script (`create_sample_data.py`) to create a sample database:
+
+   .. code-block:: python
+
+       import sqlite3
+
+       # Connect to the database (or create it if it doesn't exist)
+       conn = sqlite3.connect('sample_database.db')
+       cursor = conn.cursor()
+
+       # Create tables
+       cursor.execute('''
+       CREATE TABLE example_table (
+           id INTEGER PRIMARY KEY,
+           name TEXT,
+           value TEXT
+       )
+       ''')
+
+       cursor.execute('''
+       CREATE TABLE party_table (
+           id INTEGER PRIMARY KEY,
+           name TEXT,
+           identifier TEXT,
+           role TEXT
+       )
+       ''')
+
+       # Insert sample data into example_table
+       cursor.executemany('''
+       INSERT INTO example_table (name, value) VALUES (?, ?)
+       ''', [
+           ('sample1', 'value1'),
+           ('sample2', 'value2'),
+           ('sample3', 'value3'),
+       ])
+
+       # Insert sample data into party_table
+       cursor.executemany('''
+       INSERT INTO party_table (name, identifier, role) VALUES (?, ?, ?)
+       ''', [
+           ('party1', 'id1', 'buyer'),
+           ('party2', 'id2', 'supplier'),
+           ('party3', 'id3', 'procuringEntity'),
+       ])
+
+       # Commit and close
+       conn.commit()
+       conn.close()
+
+   Run the script to create `sample_database.db`.
+
+2. **Create a Sample Configuration File:**
+
+   Create a `sample_config.toml` file with the following content:
+
+   .. code-block:: toml
+
+       [datasource]
+       connection = 'sample_database.db'
+
+       [mapping]
+       file = 'mapping.xlsx'
+       ocid_prefix = 'ocds-123abc'
+       force_publish = true
+       selector = '''
+       SELECT
+           example_table.id AS "example_table (id)",
+           example_table.name AS "example_table (name)",
+           example_table.value AS "example_table (value)",
+           party_table.name AS "party_table (name)",
+           party_table.identifier AS "party_table (identifier)",
+           party_table.role AS "party_table (role)"
+       FROM example_table
+       JOIN party_table ON example_table.id = party_table.id;
+       '''
+
+       [publishing]
+       publisher = 'Sample Publisher'
+       version = '1.1'
+
+       [output]
+       directory = 'output'
+
+3. **Prepare the Mapping File:**
+
+   Use the following configuration for `mapping.xlsx` based on the `ocds field level mapping` template:
+
+   **General Sheet:**
+
+   .. code-block:: text
+
+       | Title         | Description   | Path             | Status   | Mapping                             | Comment
+       |---------------|---------------|------------------|----------|------------------------------------ | -------
+       | OCID          | unique ID     | ocid             | Required | example_table (id)                 | -
+       | Party ID      | Party ID      | parties/id       | Optional | party_table (identifier)           | -
+       | Party Name    | Party Name    | parties/name     | Optional | party_table (name)
+       | Role          | Role          | parties/roles    | Required | party_table (role)                 | -
+
+
+   **Tender Sheet:**
+
+   .. code-block:: text
+
+       id  | Title          | Description  | Path              | Status   | Mapping                             | Comment
+       ----|----------------|--------------|-------------------|----------|------------------------------------ | -------
+       1   | Tender Title   | Tender title | tender/title      | Optional | example_table (name)               | -
+       2   | Value          | Tender value | tender/value/amount | Optional | example_table (value)             | -
+
+
+Running the Transformation
+--------------------------
+
+Run the transformation using the CLI:
+
+.. code-block:: sh
+
+    nightingale --config sample_config.toml --package --loglevel DEBUG
+
+This will produce an output file in the `output` directory.
+
+Mapping Configuration
+----------------------
+
+Field-level mapping is specified in the `mapping.xlsx` file. It is formed from stardard `"OCDS Field Level Mapping template" <https://www.open-contracting.org/resources/ocds-field-level-mapping-template/>`_.
+For more information about how to fill the mapping file, refer to the `OCDS Field Level Mapping template guidance <https://www.open-contracting.org/resources/ocds-1-1-mapping-template-guidance/>`_.
+
+Here the bried description of the columns from mapping sheets in the mapping file:
+
+    * **Path**: The path in the OCDS release schema where the field value should be placed.
+    * **Title**: A human-readable title for the field.
+    * **Description**: A description of what the field represents.
+    * **Mapping**: The field in the source data that maps to the OCDS path.
+
+Understanding these mappings will help you configure the transformation correctly for your data.