Merge branch 'next' into next-2.0

.gitignore

@@ -5,6 +5,7 @@ creds.env
 nautobot_golden_config/transposer.py
 docker-compose.override.yml
 packages/
+invoke.yml
 
 # Ansible Retry Files
 *.retry

development/Dockerfile

@@ -64,7 +64,7 @@ RUN pip show nautobot | grep "^Version: " | sed -e 's/Version: /nautobot==/' > c
 # We can't use the entire freeze as it takes forever to resolve with rigidly fixed non-direct dependencies,
 #   especially those that are only direct to Nautobot but the container included versions slightly mismatch
 RUN poetry export -f requirements.txt --without-hashes --output poetry_freeze_base.txt
-RUN poetry export -f requirements.txt --dev --without-hashes --output poetry_freeze_all.txt
+RUN poetry export -f requirements.txt --with dev --without-hashes --output poetry_freeze_all.txt
 RUN sort poetry_freeze_base.txt poetry_freeze_all.txt | uniq -u > poetry_freeze_dev.txt
 
 # Install all local project as editable, constrained on Nautobot version, to get any additional

development/docker-compose.base.yml

@@ -21,8 +21,9 @@ services:
         condition: "service_started"
       db:
         condition: "service_healthy"
-    <<: *nautobot-build
-    <<: *nautobot-base
+    <<:
+      - *nautobot-build
+      - *nautobot-base
   worker:
     entrypoint:
       - "sh"

docs/user/app_feature_config_plans.md

@@ -0,0 +1,66 @@
+# Navigating Config Plans
+
+The natural progression for the Golden Config application is providing the ability to execute config deployments. One specific example is to work toward making one or more devices configuration compliant. To aid in this effort, the Golden Config application has the ability to generate plans containing sets of configuration commands from various sources with the intent of deploying them to devices.
+
+The current sources of these plans (i.e. plan types) are as follows:
+
+- The **Intended** configuration(s) of Compliance Feature(s)
+- The **Missing** configuration(s) of Compliance Feature(s)
+- The **Remediation** configuration(s) of Compliance Feature(s) (*)
+- A **Manual** set of configuration commands
+
+!!! note
+    The Intended, Missing and Remediation configuration come from the [Configuration Compliance](./app_feature_compliance.md#compliance-details-view) object that is created when you run the [Perform Configuration Compliance Job](./app_feature_compliance.md#starting-a-compliance-job).
+
+Much like a Configuration Compliance object, each Config Plan is tied directly to a single Device.
+
+## Viewing a Config Plan
+
+You can view a plan by navigating to **Golden Config -> Config Plans** and choosing a generated plan from the list. A Config Plan comprises of the following fields:
+
+- **Device**: The device the plan is to be deployed to.
+- **Date Created**: The date the plan was generated.
+- **Plan Type**: The type of plan used to generate it.
+- **Config Set**: The set of commands to be deployed.
+- **Features** (If Applicable): The Compliance Feature(s) the config set was generated from.
+- **Change Control ID** (Optional): A text field that be used for grouping and filtering plans.
+- **Change Control URL** (Optional): A URL field that can be used to link to an external system tracking change controls.
+- **Job Result**: The Job that generated the plan(s).
+- **Status**: The status of the plan.
+
+![Config Plan View](../images/config_plan-view.png)
+
+## Generating Config Plans
+
+In order to generate a plan, navigate to **Golden Config -> Config Plans** and hit the **Add** button. After choosing the type of plan you want to generate, you can then filter the list of devices you want to generate a Config Plan for by selecting either the list of devices themselves or a by choosing one or more related items such as Location or Status. If you select a plan type that is derived from a Configuration Compliance object, you will have the ability to only generate plans for one or more features, but selecting no features will generate plans for all applicable features.
+
+In addition, you have the ability to specify a Change Control ID & URL that can be associated with all of the plans that will be generated. This can come in handy when it comes to filtering the list of plans to ultimately deploy.
+
+Once you have selected the appropriate options, you can click the **Generate** button which will start a Job to generate the plans.
+
+### Screenshots
+
+![Config Plan Generate Missing](../images/config_plan-generate-missing.png)
+
+![Config Plan Generate Filters](../images/config_plan-generate-filters.png)
+
+![Config Plan Generate Manual](../images/config_plan-generate-manual.png)
+
+### Generating Config Plans via API
+
+The HTTP(S) POST method is not currently enabled for the Config Plan serializer to create plans directly via API. Instead you may run the **GenerateConfigPlans** Job directly via the `plugins/nautobot_golden_config.jobs/GenerateConfigPlans` API endpoint.
+
+## Editing a Config Plan
+
+After a Config Plan is generated you have the ability to edit (or bulk edit) the following fields:
+
+- Change Control ID
+- Change Control URL
+- Status
+- Notes
+- Tags
+
+!!! note
+    You will not be able to modify the Config Set after generation. If it does not contain the desired commands, you will need to delete the plan and recreate it after ensuring the source of the generated commands has been updated.
+
+![Config Plan Edit](../images/config_plan-edit.png)

docs/user/app_feature_remediation.md

@@ -0,0 +1,69 @@
+# Navigating Configuration Remediation
+
+Automated network configuration remediation is a systematic approach that leverages technology and processes to address and rectify configuration issues in network devices. 
+It involves the use of the Golden Configuration plugin to understand the current configuration state, compare it against the intended configuration state, and automatically generate remediation data.
+Automated network configuration remediation improves efficiency by eliminating manual efforts and reducing the risk of human errors. It enables rapid response to security vulnerabilities, minimizes downtime, and enhances compliance with regulatory and industry standards.
+
+
+The current sources of data to generate remediating configuration are as follows:
+
+- The **Intended** configuration of a specific Compliance Feature
+- The **Missing** configuration of a specific Compliance Feature
+- The **Extra** configuration of a specific Compliance Feature
+
+Based on this information, Golden Configuration will create a remediating configuration (if enabled for that particular platform and compliance feature). This configuration snippet will be represented as a "Remediating Configuration" field in the compliance detailed view:
+
+- The **Remediation** configuration of a specific Compliance Feature
+
+
+!!! note
+    The Intended, Missing and Extra configuration come from the [Configuration Compliance](./app_feature_compliance.md#compliance-details-view) object that is created when you run the [Perform Configuration Compliance Job](./app_feature_compliance.md#starting-a-compliance-job).
+
+
+## Setting up Configuration Remediation
+
+The type of remediation to be performed in a particular platform is defined by navigating to **Golden Config -> Remediation Settings**.
+Network device operating systems (Nautobot Platforms) can consume two different types of remediation, namely:
+
+- **HIERCONFIG remediation (CLI - hierarchical)**
+- **Custom Remediation**
+
+![Remediation Platform Settings](../images/remediation_settings_per_platform.png)
+
+### Hier Config Remediation Type
+
+Hier Config is a python library that is able to take a running configuration of a network device, compare it to its intended configuration, and build the remediation steps necessary to bring a device into spec with its intended configuration. Hier Config has been used extensively on:
+
+- Cisco IOS
+- Cisco IOSXR
+- Cisco NXOS
+- Arista EOS
+- Ruckus FastIron
+
+However, any Network Operating System (NOS) that utilizes a CLI syntax that is structured in a similar fashion to Cisco IOS should work mostly out of the box.
+Default Hier config options can be used or customized on a per platform basis, as shown below:
+
+![Hier Options Customization](../images/remediation_hier_edit_options.png)
+
+For additional information on how to customize Hier Config options, please refer to the Hierarchical Configuration development guide:
+https://netdevops.io/hier_config/advanced-topics/
+
+### Custom Config Remediation Type
+
+When a Network Operating System delivers configuration data in a format that is not CLI/Hierarchical, we can still perform remediation by using the Custom Remediation options. Custom Remediation is defined within a Python function that takes as input a Configuration Compliance object and returns a Remediation Field.
+Custom remediation performs a call to the remediation function every time a Compliance Job runs. Custom Remediation allows the user to control the configuration comparison process (between intended and actual configuration) and use additional Nautobot or external data to produce the remediation plan. Custom remediation functions need to be defined in PLUGIN_CONFIG for `nautobot_plugin_golden_config` the nautobot_config.py file, as show below:
+
+![Custom Remediation Function Setup](../images/remediation_custom_function_setup.png)
+
+## Enabling Configuration Remediation
+
+Once remediation settings are configured for a particular platform, remediation can be enabled on a per compliance rule basis. In order to enable configuration remediation for a particular rule, navigate to **Golden Config -> Compliance Rules**, and choose a rule for a platform that has remediation settings set up. Edit the compliance rule and check the box "Enable Remediation". This action effectively enables remediation for that particular Platform/Feature pair.
+
+![Enable Configuration Remediation per Feature](../images/remediation_enable_compliance_rule_feature.png)
+
+
+## Validating Configuration Remediation
+
+Once remediation is configured for a particular Platform/Feature pair, it is possible to validate remediation operations by running a compliance job. Navigate to **Jobs -> Perform Configuration Compliance** and run a compliance job for a device that has remediation enabled. Verify that remediation data has been generated by navigating to **Golden Config -> Config Compliance**, select the device and check the compliance status for the feature with remediation enabled and the "Remediating Configuration" field, as shown below:
+
+![Validate Configuration Remediation](../images/remediation_validate_feature.png)

mkdocs.yml

@@ -104,6 +104,8 @@ nav:
       - Navigate Intended: "user/app_feature_intended.md"
       - Navigate SoT Agg: "user/app_feature_sotagg.md"
       - Navigate Configuration Post-Processing: "user/app_feature_config_postprocessing.md"
+      - Navigate Config Plans: "user/app_feature_config_plans.md"
+      - Navigate Remediation: "user/app_feature_remediation.md"
       - Getting Started: "user/app_getting_started.md"
       - Frequently Asked Questions: "user/app_faq.md"
       - External Interactions: "user/app_external_interactions.md"

nautobot_golden_config/__init__.py

@@ -7,6 +7,7 @@
 
 from jinja2 import StrictUndefined
 from django.db.models.signals import post_migrate
+from nautobot.core.signals import nautobot_database_ready
 from nautobot.extras.plugins import PluginConfig
 
 
@@ -43,7 +44,15 @@ def ready(self):
         """Register custom signals."""
         from nautobot_golden_config.models import ConfigCompliance  # pylint: disable=import-outside-toplevel
 
-        from .signals import config_compliance_platform_cleanup  # pylint: disable=import-outside-toplevel
+        # pylint: disable=import-outside-toplevel
+        from .signals import (
+            config_compliance_platform_cleanup,
+            post_migrate_create_statuses,
+            post_migrate_create_job_button,
+        )
+
+        nautobot_database_ready.connect(post_migrate_create_statuses, sender=self)
+        nautobot_database_ready.connect(post_migrate_create_job_button, sender=self)
 
         super().ready()
         post_migrate.connect(config_compliance_platform_cleanup, sender=ConfigCompliance)

nautobot_golden_config/api/serializers.py

@@ -2,10 +2,16 @@
 # pylint: disable=too-many-ancestors
 from rest_framework import serializers
 
+from nautobot.apps.api import WritableNestedSerializer
+from nautobot.extras.api.fields import StatusSerializerField
+from nautobot.extras.api.serializers import TaggedObjectSerializer
+from nautobot.extras.api.nested_serializers import NestedDynamicGroupSerializer
 from nautobot.extras.api.mixins import TaggedModelSerializerMixin
+from nautobot.extras.models import Status
+from nautobot.dcim.api.nested_serializers import NestedDeviceSerializer
 from nautobot.dcim.api.serializers import DeviceSerializer
 from nautobot.dcim.models import Device
-from nautobot.extras.api.serializers import NautobotModelSerializer
+from nautobot.extras.api.serializers import NautobotModelSerializer, StatusModelSerializerMixin
 
 
 from nautobot_golden_config import models
@@ -125,3 +131,45 @@ def get_config(self, obj):
 
         config_details = models.GoldenConfig.objects.get(device=obj)
         return get_config_postprocessing(config_details, request)
+
+
+class RemediationSettingSerializer(NautobotModelSerializer, TaggedObjectSerializer):
+    """Serializer for RemediationSetting object."""
+
+    url = serializers.HyperlinkedIdentityField(
+        view_name="plugins-api:nautobot_golden_config-api:remediationsetting-detail"
+    )
+
+    class Meta:
+        """Set Meta Data for RemediationSetting, will serialize all fields."""
+
+        model = models.RemediationSetting
+        choices_fields = ["remediation_type"]
+        fields = "__all__"
+
+
+class ConfigPlanSerializer(NautobotModelSerializer, TaggedObjectSerializer, StatusModelSerializerMixin):
+    """Serializer for ConfigPlan object."""
+
+    url = serializers.HyperlinkedIdentityField(view_name="plugins-api:nautobot_golden_config-api:configplan-detail")
+    device = NestedDeviceSerializer(required=False)
+    status = StatusSerializerField(required=False, queryset=Status.objects.all())
+
+    class Meta:
+        """Set Meta Data for ConfigPlan, will serialize all fields."""
+
+        model = models.ConfigPlan
+        fields = "__all__"
+        read_only_fields = ["device", "plan_type", "feature", "config_set"]
+
+
+class NestedConfigPlanSerializer(WritableNestedSerializer):
+    """Nested serializer for ConfigPlan object."""
+
+    url = serializers.HyperlinkedIdentityField(view_name="plugins-api:nautobot_golden_config-api:configplan-detail")
+
+    class Meta:
+        """Set Meta Data for ConfigPlan, will serialize brief fields."""
+
+        model = models.ConfigPlan
+        fields = ["id", "url", "device", "plan_type"]

nautobot_golden_config/api/urls.py

@@ -14,7 +14,9 @@
 router.register("golden-config-settings", views.GoldenConfigSettingViewSet)
 router.register("config-remove", views.ConfigRemoveViewSet)
 router.register("config-replace", views.ConfigReplaceViewSet)
+router.register("remediation-setting", views.RemediationSettingViewSet)
 router.register("config-postprocessing", views.ConfigToPushViewSet)
+router.register("config-plan", views.ConfigPlanViewSet)
 urlpatterns = router.urls
 urlpatterns.append(
     path(

nautobot_golden_config/api/views.py

@@ -114,3 +114,22 @@ class ConfigToPushViewSet(mixins.RetrieveModelMixin, viewsets.GenericViewSet):
     permission_classes = [IsAuthenticated & ConfigPushPermissions]
     queryset = Device.objects.all()
     serializer_class = serializers.ConfigToPushSerializer
+
+
+class RemediationSettingViewSet(NautobotModelViewSet):  # pylint:disable=too-many-ancestors
+    """API viewset for interacting with RemediationSetting objects."""
+
+    queryset = models.RemediationSetting.objects.all()
+    serializer_class = serializers.RemediationSettingSerializer
+    filterset_class = filters.RemediationSettingFilterSet
+
+
+class ConfigPlanViewSet(NautobotModelViewSet):  # pylint:disable=too-many-ancestors
+    """API viewset for interacting with ConfigPlan objects."""
+
+    queryset = models.ConfigPlan.objects.all()
+    serializer_class = serializers.ConfigPlanSerializer
+    filterset_class = filters.ConfigPlanFilterSet
+
+    # Disabling POST as these should only be created via Job.
+    http_method_names = ["get", "put", "patch", "delete", "head", "options"]

nautobot_golden_config/choices.py

@@ -12,3 +12,31 @@ class ComplianceRuleConfigTypeChoice(ChoiceSet):
         (TYPE_CLI, "CLI"),
         (TYPE_JSON, "JSON"),
     )
+
+
+class RemediationTypeChoice(ChoiceSet):
+    """Choiceset used by RemediationSetting."""
+
+    TYPE_HIERCONFIG = "hierconfig"
+    TYPE_CUSTOM = "custom_remediation"
+
+    CHOICES = (
+        (TYPE_HIERCONFIG, "HIERCONFIG"),
+        (TYPE_CUSTOM, "CUSTOM_REMEDIATION"),
+    )
+
+
+class ConfigPlanTypeChoice(ChoiceSet):
+    """Choiceset used by ConfigPlan."""
+
+    TYPE_INTENDED = "intended"
+    TYPE_MISSING = "missing"
+    TYPE_REMEDIATION = "remediation"
+    TYPE_MANUAL = "manual"
+
+    CHOICES = (
+        (TYPE_INTENDED, "Intended"),
+        (TYPE_MISSING, "Missing"),
+        (TYPE_REMEDIATION, "Remediation"),
+        (TYPE_MANUAL, "Manual"),
+    )

nautobot_golden_config/filter_extensions.py

@@ -0,0 +1,23 @@
+"""Custom filter to extend base API for filterform use case."""
+import django_filters
+from nautobot.apps.filters import FilterExtension
+
+
+def config_plan_null_search(queryset, name, value):  # pylint: disable=unused-argument
+    """Query to ensure config plans are not empty."""
+    return queryset.filter(config_plan__isnull=False).distinct()
+
+
+class JobResultFilterExtension(FilterExtension):
+    """Filter provided to be used in select2 query for only jobs that were used by ConfigPlan."""
+
+    model = "extras.jobresult"
+
+    filterset_fields = {
+        "nautobot_golden_config_config_plan_null": django_filters.BooleanFilter(
+            label="Is FK to ConfigPlan Model", method=config_plan_null_search
+        )
+    }
+
+
+filter_extensions = [JobResultFilterExtension]