Add auditree empty evidence check (#63)

.pre-commit-config.yaml

@@ -1,13 +1,13 @@
 repos:
 -   repo: https://github.com/pre-commit/mirrors-yapf
-    rev: v0.30.0
+    rev: v0.31.0
     hooks:
     -   id: yapf
         args: [--in-place, --parallel, --recursive, --style, .yapf-config]
         files: "^(arboretum|test)"
         stages: [commit]
 -   repo: https://gitlab.com/pycqa/flake8
-    rev: 3.8.4
+    rev: 3.9.1
     hooks:
     -   id: flake8
         args: [

CHANGES.md

@@ -1,3 +1,7 @@
+# [0.13.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.13.0)
+
+- [ADDED] Auditree empty evidence check has been added.
+
 # [0.12.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.12.0)
 
 - [ADDED] Github org integrity teams fetcher functionality added to Github org integrity permissions fetcher.

arboretum/__init__.py

@@ -14,4 +14,4 @@
 # limitations under the License.
 """Arboretum - Checking your compliance & security posture, continuously."""
 
-__version__ = '0.12.0'
+__version__ = '0.13.0'

arboretum/auditree/README.md

@@ -432,6 +432,50 @@ execution.  The default threshold is 30 days beyond the time to live (TTL) setti
    from arboretum.auditree.checks.test_abandoned_evidence import AbandonedEvidenceCheck
    ```
 
+### Empty Evidence
+
+* Class: [EmptyEvidenceCheck][check-empty-evidence]
+* Purpose: For every piece of evidence that has no content a failure is generated
+and reported.
+* Behavior: Performs a check that validates the content of evidence is not empty.
+Empty evidence content is based on an evidence object's `is_empty` property which
+can vary across evidences.  But the default `is_empty` criteria is as follows:
+   * Content is all whitespace.
+   * In the case of JSON, content is an empty dictionary or list (`{}`, `[]`).
+* Evidence depended upon:
+   * This check does not depend on any evidence specifically.  It acts on **all**
+   evidence contained within an evidence locker.
+* Configuration elements:
+   * `org.auditree.empty_evidence.exceptions`
+      * Optional
+      * List where the list elements are the relative evidence locker paths to
+      evidence files.
+      * Use if looking to exclude specific evidence files from being flagged as
+      failures.  All "exceptions" will still appear on the report and will be
+      treated as warnings rather than failures.
+* Example (optional) configuration:
+
+   ```json
+   {
+     "org": {
+       "auditree": {
+         "empty_evidence": {
+           "exceptions": [
+             "raw/path/to-evidence.json",
+             "raw/path/to-evidence-2.json"
+           ]
+         }
+       }
+     }
+   }
+   ```
+
+* Import statement:
+
+   ```python
+   from arboretum.auditree.checks.test_empty_evidence import EmptyEvidenceCheck
+   ```
+
 ### Compliance Configuration
 
 * Class: [ComplianceConfigCheck][check-compliance-config]
@@ -698,6 +742,7 @@ getting new commits.  This check validates that.
 [fetch-filepath-commits]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/fetchers/github/fetch_filepath_commits.py
 [fetch-branch-protection]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/fetchers/github/fetch_branch_protection.py
 [check-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_abandoned_evidence.py
+[check-empty-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_empty_evidence.py
 [check-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_compliance_config.py
 [check-python-packages]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_python_packages.py
 [check-locker-integrity]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_locker_repo_integrity.py

arboretum/auditree/checks/test_empty_evidence.py

@@ -0,0 +1,76 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2021 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Evidence locker empty evidences check."""
+from compliance.check import ComplianceCheck
+from compliance.evidence import DAY, ReportEvidence
+
+
+class EmptyEvidenceCheck(ComplianceCheck):
+    """
+    Empty evidence check.
+
+    This check finds all evidence in the evidence locker that has no content.
+    """
+
+    @property
+    def title(self):
+        """
+        Return the title of the checks.
+
+        :returns: the title of the checks
+        """
+        return 'Empty Evidence'
+
+    @classmethod
+    def setUpClass(cls):
+        """Initialize the check object with configuration settings."""
+        cls.config.add_evidences(
+            [
+                ReportEvidence(
+                    'empty_evidence.md',
+                    'auditree',
+                    DAY,
+                    'Evidence locker empty evidence report.'
+                )
+            ]
+        )
+        return cls
+
+    def test_empty_evidence(self):
+        """Check for evidence that has no content."""
+        exceptions = self.config.get(
+            'org.auditree.empty_evidence.exceptions', []
+        )
+        for ev_path in self.locker.get_empty_evidences():
+            if ev_path not in exceptions:
+                self.add_failures('Empty Evidence', f'`{ev_path}`')
+            else:
+                self.add_warnings('Expected Empty Evidence', f'`{ev_path}`')
+
+    def get_reports(self):
+        """
+        Provide the check report name.
+
+        :returns: the report(s) generated for this check.
+        """
+        return ['auditree/empty_evidence.md']
+
+    def get_notification_message(self):
+        """
+        Empty evidence check notifier.
+
+        :returns: notification dictionary.
+        """
+        return {'body': None}