Deploy Readthedocs (#39)

* Init readthedocs page

* Update docstring and fix some bugs

.readthedocs.yaml

@@ -0,0 +1,24 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+sphinx:
+   configuration: docs/source/conf.py
+
+
+python:
+  install:
+    # install itself with pip install .
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs

CONTRIBUTING.md

@@ -17,6 +17,8 @@ Pre-commit will automatically format the code before each commit, It can also be
 pre-commit run --all-files
 ```
 
+Comment style is [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings).
+
 ## Install Locally with Test Dependencies
 
 ```bash
@@ -34,6 +36,35 @@ pytest --cov=duetector # Generate coverage reports
 
 Run unit-test before PR, **ensure that new features are covered by unit tests**
 
+## Generating config
+
+Use script to generate config after add tracer/filter...
+
+```bash
+python duetector/tools/config_generator.py
+```
+
+## Build Docs
+
+Install docs dependencies
+
+```bash
+pip install -e .[docs]
+```
+
+Build docs
+
+```bash
+make clean && make html
+```
+
+Use [start-docs-host.sh](dev-tools/start-docs-host.sh) to deploy a local http server to view the docs
+
+```bash
+cd ./dev-tools && ./start-docs-host.sh
+```
+Access `http://localhost:8080` for docs.
+
 ## Typing
 
 (Optional, python<=3.10) Use [pytype](https://github.com/google/pytype) to check typed

README.md

@@ -1,6 +1,7 @@
 <h2 align="center">duetector🔍: 支持eBPF的可扩展数据使用探测器</h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>
+<a href='https://duetector.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/duetector/badge/?version=latest' alt='Documentation Status' /></a>
 <a href="https://results.pre-commit.ci/latest/github/hitsz-ids/duetector/main"><img alt="pre-commit.ci status" src="https://results.pre-commit.ci/badge/github/hitsz-ids/duetector/main.svg"></a>
 <a href="https://github.com/hitsz-ids/duetector/blob/main/LICENSE"><img alt="LICENSE" src="https://img.shields.io/github/license/hitsz-ids/duetector"></a>
 <a href="https://github.com/hitsz-ids/duetector/releases/"><img alt="Releases" src="https://img.shields.io/github/v/release/hitsz-ids/duetector"></a>
@@ -157,7 +158,7 @@ Commands:
 
 ## API文档与配置文档
 
-WIP 这一部分内容是PIP相关的，目前还没有完成，完成后将包括可配置的类的内容，以及如何使用duetector作为PIP的内容。
+我们在readthedocs上为开发者和用户提供了API与配置文档，你可以在[这里](https://duetector.readthedocs.io/)查看
 
 ## 维护者
 

README_en.md

@@ -1,8 +1,11 @@
 <h2 align="center">duetector🔍: Data Usage Extensible detector </h2>
 <p align="center">
 <a href="https://github.com/hitsz-ids/duetector/actions"><img alt="Actions Status" src="https://github.com/hitsz-ids/duetector/actions/workflows/python-package.yml/badge.svg"></a>
+<a href='https://duetector.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/duetector/badge/?version=latest' alt='Documentation Status' /></a>
+<a href="https://results.pre-commit.ci/latest/github/hitsz-ids/duetector/main"><img alt="pre-commit.ci status" src="https://results.pre-commit.ci/badge/github/hitsz-ids/duetector/main.svg"></a>
 <a href="https://github.com/hitsz-ids/duetector/blob/main/LICENSE"><img alt="LICENSE" src="https://img.shields.io/github/license/hitsz-ids/duetector"></a>
 <a href="https://github.com/hitsz-ids/duetector/releases/"><img alt="Releases" src="https://img.shields.io/github/v/release/hitsz-ids/duetector"></a>
+<a href="https://github.com/hitsz-ids/duetector/releases/"><img alt="Pre Releases" src="https://img.shields.io/github/v/release/hitsz-ids/duetector?include_prereleases&label=pre-release&logo=github"></a>
 <a href="https://github.com/hitsz-ids/duetector"><img alt="Last Commit" src="https://img.shields.io/github/last-commit/hitsz-ids/duetector"></a>
 <a href="https://github.com/hitsz-ids/duetector"><img alt="Python version" src="https://img.shields.io/pypi/pyversions/duetector"></a>
 </p>
@@ -152,7 +155,7 @@ More documentation and examples can be found [here](. /docs/).
 
 ## API documentation
 
-WIP
+See [docs of duetector](https://duetector.readthedocs.io/)
 
 ## Maintainers
 

dev-tools/start-docs-host.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+# Start a nginx server for host ../docs/build/html
+# This is useful for testing the docs locally
+
+set -e
+docker run --rm \
+-it \
+-p 8080:80 \
+-v $(pwd)/../docs/build/:/usr/share/nginx/:ro \
+nginx

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/source/cli/daemon.rst

@@ -0,0 +1,3 @@
+.. click:: duetector.cli.daemon:cli
+   :prog: duectl-daemon
+   :nested: full

docs/source/cli/index.rst

@@ -0,0 +1,14 @@
+duetector.cli
+=========================================
+
+``duectl``: CLI for Start Monitor, generate config.
+
+``duectl-daemon``: Allow to run as daemon, and run as a service.
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Reference API docs:
+
+   duectl <main>
+   duectl-daemon <daemon>

docs/source/cli/main.rst

@@ -0,0 +1,3 @@
+.. click:: duetector.cli.main:cli
+   :prog: duectl
+   :nested: full

docs/source/collectors/base.rst

@@ -0,0 +1,16 @@
+Collector
+==================
+
+Base class for all collectors.
+
+.. autoclass:: duetector.collectors.base.Collector
+   :members:
+   :undoc-members:
+   :private-members:
+
+
+.. autoclass:: duetector.collectors.base.DequeCollector
+   :members:
+   :undoc-members:
+   :private-members:
+   :show-inheritance:

docs/source/collectors/db.rst

@@ -0,0 +1,10 @@
+DBCollector
+==================
+
+DBCollector use :doc:`SessionManager </db>` for config the db engine and create sessions.
+
+.. autoclass:: duetector.collectors.db.DBCollector
+   :members:
+   :undoc-members:
+   :private-members:
+   :show-inheritance:

docs/source/collectors/index.rst

@@ -0,0 +1,13 @@
+Collector
+=====================================
+
+Collector will convert ``data_t`` to :doc:`Tracking <models>`
+and store them in a somewhere.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Avaliable Collector
+
+   Base Collectors <base>
+   DB Collectors <db>
+   Data Models <models>

docs/source/collectors/models.rst

@@ -0,0 +1,7 @@
+Models for collectors
+====================================
+
+.. autoclass:: duetector.collectors.models.Tracking
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/conf.py

@@ -0,0 +1,67 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath(".."))
+
+# -- Project information -----------------------------------------------------
+
+project = "duetector"
+copyright = "2023, hitsz-ids"
+author = "hitsz-ids"
+
+# The full version, including alpha/beta/rc tags
+from duetector import __version__
+
+release = __version__
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.autosectionlabel",
+    "sphinx_click",
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = "en"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_rtd_theme"
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]

docs/source/config.rst

@@ -0,0 +1,6 @@
+Config utilities
+=========================================
+.. automodule:: duetector.config
+   :members:
+   :undoc-members:
+   :inherited-members:

docs/source/db.rst

@@ -0,0 +1,6 @@
+Database utilities
+=========================================
+.. automodule:: duetector.db
+   :members:
+   :undoc-members:
+   :inherited-members:

docs/source/exceptions.rst

@@ -0,0 +1,6 @@
+Exceptions
+=========================================
+.. automodule:: duetector.exceptions
+   :members:
+   :undoc-members:
+   :inherited-members:

docs/source/filters/base.rst

@@ -0,0 +1,10 @@
+BaseFilter
+==================
+
+Base class for all filters.
+
+
+.. autoclass:: duetector.filters.base.Filter
+   :members:
+   :undoc-members:
+   :private-members:

docs/source/filters/index.rst

@@ -0,0 +1,11 @@
+Filter
+=====================================
+
+``Filter`` will filter the data based on the given criteria.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Avaliable Filter
+
+   Base Filter <base>
+   Pattern Filter <pattern>

docs/source/filters/pattern.rst

@@ -0,0 +1,10 @@
+PatternFilter
+==================
+
+``PatternFilter`` support regex pattern to filter data.
+
+.. autoclass:: duetector.filters.pattern.PatternFilter
+   :members:
+   :undoc-members:
+   :private-members:
+   :show-inheritance:

docs/source/index.rst

@@ -0,0 +1,36 @@
+
+Welcome to duetector's documentation!
+=========================================
+
+duetector🔍 is an extensible data usage control detector that provides support for data usage control by probing for data usage behavior in the Linux kernel(based on eBPF).
+
+For more information, please visit the `project homepage <https://github.com/hitsz-ids/duetector>`_.
+
+Reference
+========================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Reference Documentation:
+
+   CLI <cli/index>
+
+   Monitors <monitors/index>
+   Managers <managers/index>
+
+   Tracers <tracers/index>
+   Filters <filters/index>
+   Collectors <collectors/index>
+
+   Exceptions <exceptions>
+
+   Database utilities <db>
+   Config utilities <config>
+   Tools utilities <tools/index>
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`search`

docs/source/managers/collector.rst

@@ -0,0 +1,13 @@
+CollectorManager
+===========================================
+
+
+.. autodata:: duetector.managers.collector.PROJECT_NAME
+.. autofunction:: duetector.managers.collector.init_collector
+
+
+.. autoclass:: duetector.managers.CollectorManager
+   :members:
+   :undoc-members:
+   :inherited-members:
+   :show-inheritance: