Intro analyzer for trackings (#48)

* Update design docs * Prepare class and docs * Add inspect for db and some improve * Support query and inspect trackings * Convert timestamp to datetime * More accuracy of datettime * More on analyzer & Resolving circular dependencies * Intro autodoc-pydantic * Fix docs and more on base * Support detailed query * Support orderby in query * Add with_details on brief * Support tracers, collector_ids and distinct
hitsz-ids · Sep 12, 2023 · a1dd6e4 · a1dd6e4
1 parent 9715b79
commit a1dd6e4
Show file tree

Hide file tree

Showing 47 changed files with 1,076 additions and 170 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -63,6 +63,7 @@ Use [start-docs-host.sh](dev-tools/start-docs-host.sh) to deploy a local http se
 ```bash
 cd ./dev-tools && ./start-docs-host.sh
 ```
+
 Access `http://localhost:8080` for docs.
 
 ## Typing
@@ -72,3 +73,11 @@ Access `http://localhost:8080` for docs.
 ```
 pytype ./duetector
 ```
+
+
+## Contributing a new tracer/filter/collector
+
+1. Create a new file in `duetector/tracer`, `duetector/filter` or `duetector/collector` directory, with the name `{name}.py`
+2. Implement the new tracer/filter/collector
+3. Add the new tracer/filter/collector to `registers` list in `duetector/tracer/register.py`, `duetector/filter/register.py` or `duetector/collector/register.py`
+4. Test the new tracer/filter/collector
diff --git a/docs/design/README.md b/docs/design/README.md
@@ -6,6 +6,7 @@ Key Components and Features:
 
 - [ ] **HTTP / RPC Server**: PIP Server, providing API for PDP to get data usage information.
 - [ ] **Analyzer**: Analyze data usage information and generate data usage behavior.
+  - [ ] **DBAnalyzer**: Analyze data usage information from database.
 - [X] **CLI**: CLI for administrator to manage duetector.
 - [X] **BccMonitor**: Monitor data usage behavior in kernel space. Use BCC to implement.
 - [X] **ShMonitor**: A general monitor for custom command. Polling the output of command.
@@ -32,5 +33,5 @@ Current data flow implementation:
 
 The following are not yet realized and may be subject to change.
 
-- [ ] **Collector** will get data from **Collector**.
-- [ ] **HTTP / RPC Server** will get data from **Collector**.
+- [ ] **Analyzer**'s data stracture and API.
+- [ ] **Query Service** will get data from **Analyzer**.
diff --git a/docs/design/image/architecture.png b/docs/design/image/architecture.png
diff --git a/docs/design/image/dataflow.png b/docs/design/image/dataflow.png
diff --git a/docs/design/src/architecture.drawio b/docs/design/src/architecture.drawio
diff --git a/docs/design/src/dataflow.drawio b/docs/design/src/dataflow.drawio
diff --git a/docs/source/analyzer/db.rst b/docs/source/analyzer/db.rst
@@ -0,0 +1,10 @@
+DBAnalyzer
+===============================
+
+``DBAnalyzer``
+
+.. automodule:: duetector.analyzer.db
+   :members:
+   :undoc-members:
+   :private-members:
+   :show-inheritance:
diff --git a/docs/source/analyzer/index.rst b/docs/source/analyzer/index.rst
@@ -0,0 +1,25 @@
+Analyzer
+=========================================
+
+
+.. autoclass:: duetector.analyzer.base.Analyzer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+Avaliable Analyzer
+-----------------------------------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   DB Analyzer <db>
+
+Data Models
+-----------------------------------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   Data Models <models>
diff --git a/docs/source/analyzer/models.rst b/docs/source/analyzer/models.rst
@@ -0,0 +1,7 @@
+Data Models
+===============================
+
+.. automodule:: duetector.analyzer.models
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/source/collectors/index.rst b/docs/source/collectors/index.rst
@@ -18,9 +18,19 @@ and store them in a somewhere.
    :show-inheritance:
 
 
+Avaliable Collector
+---------------------------------------------------------
+
 .. toctree::
    :maxdepth: 2
-   :caption: Avaliable Collector and Data Models
 
    DB Collectors <db>
+
+
+Data Models
+---------------------------------------------------------
+
+.. toctree::
+   :maxdepth: 2
+
    Data Models <models>
diff --git a/docs/source/collectors/models.rst b/docs/source/collectors/models.rst
@@ -1,7 +1,7 @@
 Models for collectors
 ====================================
 
-.. autoclass:: duetector.collectors.models.Tracking
+.. automodule:: duetector.collectors.models
    :members:
    :undoc-members:
    :show-inheritance:
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -37,6 +37,7 @@
     "sphinx.ext.viewcode",
     "sphinx.ext.autosectionlabel",
     "sphinx_click",
+    "sphinxcontrib.autodoc_pydantic",
 ]
 
 # Add any paths that contain templates here, relative to this directory.

diff --git a/docs/source/filters/base.rst b/docs/source/filters/base.rst
diff --git a/docs/source/filters/index.rst b/docs/source/filters/index.rst
@@ -3,9 +3,16 @@ Filter
 
 ``Filter`` will filter the data based on the given criteria.
 
+.. autoclass:: duetector.filters.base.Filter
+   :members:
+   :undoc-members:
+   :private-members:
+
+
+Avaliable Filter
+------------------------------------------------------
+
 .. toctree::
    :maxdepth: 2
-   :caption: Avaliable Filter
 
-   Base Filter <base>
    Pattern Filter <pattern>
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -15,18 +15,15 @@ Reference
 
    CLI <cli/index>
 
+   Analyzer <analyzer/index>
    Monitors <monitors/index>
    Managers <managers/index>
 
    Tracers <tracers/index>
    Filters <filters/index>
    Collectors <collectors/index>
 
-   Exceptions <exceptions>
-
-   Database utilities <db>
-   Config utilities <config>
-   Tools utilities <tools/index>
+   Utilities <utilities>
 
 
 Indices and tables

diff --git a/docs/source/managers/index.rst b/docs/source/managers/index.rst
@@ -10,9 +10,12 @@ Managers provides a way to get instances of both built-in implementations and ex
    :inherited-members:
    :show-inheritance:
 
+
+Avaliable Manager
+-------------------------------------------
+
 .. toctree::
-   :maxdepth: 1
-   :caption: Avaliable Manager
+   :maxdepth: 2
 
    Collector Manager <collector>
    Filter Manager <filter>

diff --git a/docs/source/monitors/index.rst b/docs/source/monitors/index.rst
@@ -14,9 +14,11 @@ entry point for the polling, filtering and collecting of the data.
    :show-inheritance:
 
 
+Avaliable Monitor
+-------------------------------------------
+
 .. toctree::
-   :maxdepth: 1
-   :caption: Avaliable Monitor
+   :maxdepth: 2
 
    Bcc Monitor <bcc>
    Shell Monitor <sh>
diff --git a/docs/source/tracers/index.rst b/docs/source/tracers/index.rst
@@ -1,15 +1,25 @@
 Tracer
-=========================================
+=====================================
+
+``Tracer`` will be capturing information in some way.
+:doc:`Collector </collectors/index>` will convert ``Tracer``'s  ``data_t`` to :doc:`Tracking </collectors/models>`.
+
+.. note::
+   Some filed of ``data_t`` will be converted to other more readable filed,
+   if you want to fit this feature, you should refer to :doc:`Tracking.normalize_field </collectors/models>`.
+
 
 .. automodule:: duetector.tracers
    :members:
    :undoc-members:
    :inherited-members:
 
 
+Avaliable Tracer
+--------------------------------------
+
 .. toctree::
-   :maxdepth: 1
-   :caption: Avaliable Tracer
+   :maxdepth: 2
 
    CloneTracer <clone>
    OpenTracer <openat2>

diff --git a/docs/source/utilities.rst b/docs/source/utilities.rst
@@ -0,0 +1,12 @@
+Utilities
+===============================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Utilities Documentation:
+
+   Exceptions <exceptions>
+
+   Database utilities <db>
+   Config utilities <config>
+   Tools utilities <tools/index>
diff --git a/duetector/analyzer/__init__.py b/duetector/analyzer/__init__.py
diff --git a/duetector/analyzer/base.py b/duetector/analyzer/base.py
@@ -0,0 +1,69 @@
+from datetime import datetime
+from typing import List, Optional
+
+from duetector.analyzer.models import AnalyzerBrief, Tracking
+from duetector.config import Configuable
+
+
+class Analyzer(Configuable):
+    """
+    A base class for all analyzers.
+    """
+
+    default_config = {}
+    """
+    Default config for ``Analyzer``.
+    """
+
+    config_scope = "analyzer"
+    """
+    Config scope for this analyzer.
+
+    Subclasses cloud override this.
+    """
+
+    def get_all_tracers(self) -> List[str]:
+        """
+        Get all tracers from storage.
+
+        Returns:
+            List[str]: List of tracer's name.
+        """
+        raise NotImplementedError
+
+    def get_all_collector_ids(self) -> List[str]:
+        """
+        Get all collector id from storage.
+
+        Returns:
+            List[str]: List of collector id.
+        """
+        raise NotImplementedError
+
+    def query(
+        self,
+        tracer: Optional[str] = None,
+        collector_id: Optional[str] = None,
+        start_datetime: Optional[datetime] = None,
+        end_datetime: Optional[datetime] = None,
+        start: int = 0,
+        limit: int = 20,
+    ) -> List[Tracking]:
+        """
+        Query tracking data from storage.
+        """
+        raise NotImplementedError
+
+    def brief(
+        self,
+        start_datetime: Optional[datetime] = None,
+        end_datetime: Optional[datetime] = None,
+    ) -> AnalyzerBrief:
+        """
+        Get brief of analyzer.
+        """
+        raise NotImplementedError
+
+    def analyze(self):
+        # TODO: Not design yet.
+        pass