Feature: Add REST API to render intended config Jinja templates

changes/824.added

@@ -0,0 +1 @@
+Added a REST API endpoint for Jinja template developers to render intended configurations from templates in an arbitrary git repository.

changes/824.housekeeping

@@ -0,0 +1 @@
+Updated multiple tests to use the faster `setUpTestData` instead of `setUp`. Fixed incorrect base class on `ConfigPlanTest`.

docs/user/app_feature_config_postprocessing.md

@@ -1,64 +1,64 @@
 # Navigating Configuration Post-processing
 
 !!! note
-    Current implementation **only renders the configuration to push, it doesn't update the configuration** into the target devices.
+    The current implementation **only renders the configuration for pushing and does not update the configuration** on the target devices.
 
-The intended configuration job doesn't produce a final configuration artifact (see below for reasons why). The intended configuration is the "intended" **running** configuration, because the intended configuration job generates what is in the final running configuration. This works well for the "compliance" feature, but not as well to create a configuration artifact that is ready to push.
+The intended configuration job doesn't produce a final configuration artifact (see below for reasons). The intended configuration represents the "intended" **running** configuration, as it generates what is expected to be in the final running configuration. While this approach works well for the "compliance" feature, it is less effective for creating a configuration artifact that is ready to be pushed to devices.
 
-Challenging use cases when using the running configuration as intended:
+Challenges when using the running configuration as the intended configuration:
 
-- Because the intended configuration is stored in the database, and in an external Git repository, it should **not** contain any secret.
-- The format of the running configuration is not always the same as the configuration to push, examples include:
-    - Pushing SNMPv3 configurations, which do not show up in the running config
-    - VTP configurations where the configurations is not in the running config at all
-    - Implicit configurations like a "no shutdown" on an interface
-- The configurations used to get the configuration to the intended state may require to be ordered to not cause an outage.
+- Since the intended configuration is stored in both the database and an external Git repository, it should **not** contain any secrets.
+- The format of the running configuration may differ from the configuration that needs to be pushed. Examples include:
+    - SNMPv3 configurations, which do not appear in the running configuration
+    - VTP configurations that are entirely absent from the running configuration
+    - Implicit configurations, such as "no shutdown" commands on interfaces
+- Configurations necessary to achieve the intended state may need to be ordered carefully to prevent outages.
 
-As the Golden Config application becomes more mature in delivering an all encompassing configuration management solution, it requires an advanced feature to render a configuration artifact. That artifact must be in the final format your device is expecting, from the intended configuration.
+As the Golden Config application evolves into a comprehensive configuration management solution, it requires an advanced feature to generate a configuration artifact that is in the final format expected by your device, based on the intended configuration.
 
-This is exposed via the `get_config_postprocessing()` function defined in `nautobot_golden_config.utilities.config_postprocessing`. This method takes the current configurations generated by the Golden Config intended configuration feature, and the HTTP request. This function will return the intended configuration that is **ready to push**.
+This is achieved through the `get_config_postprocessing()` function defined in `nautobot_golden_config.utilities.config_postprocessing`. This method processes the configurations generated by the Golden Config intended configuration feature, along with the HTTP request. It returns the intended configuration that is **ready to be pushed**.
 
-From the user perspective, you can retrieve this configuration via two methods:
+From a user perspective, you can retrieve this configuration using two methods:
 
-- UI: within the `Device` detail view, if the feature is enabled, a new row in the "Configuration Types" appears, and clicking the icon the new configuration will be rendered on the fly (synchronously). Check figure.
-- REST API: at the path `/api/plugins/golden-config/config-postprocessing/{device_id}` you can request the intended configuration processed, and the return payload will contain a "config" key with the rendered configuration.
+- **UI**: In the `Device` detail view, if the feature is enabled, a new row appears under "Configuration Types." Clicking the icon renders the new configuration on the fly (synchronously). See the figure below for reference.
+- **REST API**: You can request the processed intended configuration at the path `/api/plugins/golden-config/config-postprocessing/{device_id}`. The return payload will contain a "config" key with the rendered configuration.
 
 ![Configuration Postprocessing](../images/config_postprocessing_1.png)
 
-## Customize Configuration Processing
+## Customizing Configuration Processing
 
-There are two different ways to customize the default behavior of `get_config_postprocessing` method:
+There are two ways to customize the default behavior of the `get_config_postprocessing` method:
 
-- `postprocessing_callables`: is the list of **available methods** for processing the intended configuration. It contains some default implemented methods, currently `render_secrets`. But it could be extended via configuration options (see next section). The format for defining these methods is via the dotted string format that will be imported by Django. For example, the `render_secrets` is defined as `"nautobot_golden_config.utilities.config_postprocessing.render_secrets"`.
-- `postprocessing_subscribed`: is the list of **methods names** (strings) that define the **order** in the processing chain. The defined methods MUST exist in the `postprocessing_callables` list. This list can be customized via configuration options, and eventually, it could be extended to accept HTTP query parameters.
+- `postprocessing_callables`: A list of **available methods** for processing the intended configuration. It includes some default methods, such as `render_secrets`, but can be extended via configuration options (see the next section). These methods are defined using a dotted string format that Django imports. For example, `render_secrets` is defined as `"nautobot_golden_config.utilities.config_postprocessing.render_secrets"`.
+- `postprocessing_subscribed`: A list of **method names** (strings) that define the **order** in which methods are executed. The methods must exist in the `postprocessing_callables` list. This list can be customized through configuration options and could eventually accept HTTP query parameters for further customization.
 
-## Existing Default Processors
+## Default Processors
 
 ### Render Secrets
 
-The `render_secrets` function performs an extra Jinja rendering on top of an intended configuration, exposing new custom Jinja filters:
+The `render_secrets` function performs an additional Jinja rendering on the intended configuration, providing custom Jinja filters:
 
-- `get_secret_by_secret_group_name`: as the name suggests, it returns the secret_group value, for a secret type, from its `name`.
+- `get_secret_by_secret_group_name`: As the name implies, this filter returns the value of a secret group for a given secret type based on its `name`.
 
 !!! note
-    Other default Django or Netutils filters are not available in this Jinja environment. Only `encrypt_<vendor>_type5` and `encrypt_<vendor>_type7` can be used together with the `get_secret` filters.
+    Standard Django or Netutils filters are not available in this Jinja environment. Only `encrypt_<vendor>_type5` and `encrypt_<vendor>_type7` filters can be used in conjunction with the `get_secret` filters.
 
-Because this rendering is separated from the standard generation of the intended configuration, you must use the `{% raw %}` Jinja syntax to avoid being processed by the initial generation stage.
+Since this rendering occurs after the initial generation of the intended configuration, the `{% raw %}` Jinja syntax must be used to prevent premature processing.
 
-1. For example, an original template like this, `{% raw %}ppp pap sent-username {{ secrets_group["name"] | get_secret_by_secret_group_name("username")}}{% endraw %}`
-2. Produces an intended configuration as `ppp pap sent-username {{ secrets_group["name"] | get_secret_by_secret_group_name("username") }}`
-3. After the `render_secrets`, it becomes `ppp pap sent-username my_username`.
+1. For example, an original template might look like this: `{% raw %}ppp pap sent-username {{ secrets_group["name"] | get_secret_by_secret_group_name("username") }}{% endraw %}`
+2. It produces an intended configuration like this: `ppp pap sent-username {{ secrets_group["name"] | get_secret_by_secret_group_name("username") }}`
+3. After applying `render_secrets`, it becomes: `ppp pap sent-username my_username`.
 
-Notice that the `get_secret` filters take arguments. In the example, the `secret_group` name is passed, together with the type of the `Secret`. Check every signature for extra customization.
+Note that the `get_secret` filters accept arguments. In the example, the `secret_group` name is passed along with the type of secret. You can customize the signature for additional options.
 
 !!! note
-    Remember that to render these secrets, the user requesting it via UI or API, MUST have read permissions to Secrets Groups, Golden Config, and the specific Device object.
+    To render secrets, the user requesting the configuration via UI or API **must** have read permissions for Secrets Groups, Golden Config, and the specific Device object.
 
 #### Render Secrets Example
 
-This shows how Render the Secrets feature for a `Device`, for the default `Secrets Group` FK, and for custom relationships, in the example, at `Location` level.
+Here is an example of rendering secrets for a `Device`, using the default `Secrets Group` ForeignKey (FK) and custom relationships, in this case at the `Location` level.
 
-##### GraphQL query
+##### GraphQL Query
 
 ```graphql
 query ($device_id: ID!) {
@@ -89,11 +89,11 @@ Using the custom relationship at the `Location` level:
 {% raw %}{{ location["rel_my_secret_relationship_for_location"][0]["name"] | get_secret_by_secret_group_name("password") | default('no password') }}{% endraw %}
 ```
 
-This will end up rendering the secret, of type "password", for the corresponding `SecretGroup`.
+This will render the secret of type "password" for the corresponding `SecretGroup`.
 
-##### Managing errors
+##### Managing Errors
 
-Obviously, the rendering process can find multiple challenges, that are managed, and properly explained to take corrective actions:
+The rendering process may encounter issues, which are managed and properly explained to guide corrective actions:
 
 ```
 Found an error rendering the configuration to push: Jinja encountered and UndefinedError: 'None' has no attribute 'name', check the template for missing variable definitions.

docs/user/app_feature_intended.md

@@ -33,6 +33,27 @@ or
 
 In these examples, `/services.j2`, `/ntp.j2`, etc. could contain the actual Jinja code which renders the configuration for their corresponding features. Alternately, in more complex environments, these files could themselves contain only include statements in order to create a hierarchy of template files so as to keep each individual file neat and simple. Think of the main, top-level, template as an entrypoint into a hierarchy of templates. A well thought out structure to your templates is necessary to avoid the temptation to place all logic into a small number of templates. Like any code, Jinja2 functions become harder to manage, more buggy, and more fragile as you add complexity, so any thing which you can do to keep them simple will help your automation efforts.
 
+### Developing Intended Configuration Templates
+
+To help developers create the Jinja2 templates for generating the intended configuration, the app provides a REST API at `/api/plugins/golden-config/generate-intended-config/`. This API accepts two query parameters: `device_id` and `git_repository_id`. It returns the rendered configuration for the specified device using the templates from the given Git repository. This feature allows developers to test their configuration templates using a custom `GitRepository` without running a full intended configuration job.
+
+Here’s an example of how to request the rendered configuration for a device:
+
+```no-highlight
+GET /api/plugins/golden-config/generate-intended-config/?device_id=231b8765-054d-4abe-bdbf-cd60e049cd8d&git_repository_id=82c051e0-d0a9-4008-948a-936a409c654a
+```
+
+The returned response will contain the rendered configuration for the specified device. This is the intended workflow for developers:
+
+- Create a new branch in the intended configuration repository.
+- Modify the Jinja2 templates in that new branch.
+- Add a new `GitRepository` in Nautobot that points to the new branch and sync the repository.
+- Use the API to render the configuration for a device, using the new `GitRepository`.
+
+Keep in mind that Nautobot only pulls the latest branch updates when you sync the `GitRepository`. If you make changes to the branch after syncing, you'll need to sync the repository again to apply the latest updates.
+
+Note that this API is only intended to render Jinja2 templates but does not apply any [configuration post-processing](./app_feature_config_postprocessing.md).
+
 ## Adding Jinja2 Filters to the Environment.
 
 This app follows [Nautobot](https://docs.nautobot.com/projects/core/en/stable/plugins/development/#including-jinja2-filters) in relying on [django_jinja](https://niwinz.github.io/django-jinja/latest/) for customizing the Jinja2 Environment. Currently, only filters in the `django_jinja` Environment are passed along to the Jinja2 Template Environment used by Nornir to render the config template.

nautobot_golden_config/api/serializers.py

@@ -124,3 +124,10 @@ class Meta:
         model = models.ConfigPlan
         fields = "__all__"
         read_only_fields = ["device", "plan_type", "feature", "config_set"]
+
+
+class GenerateIntendedConfigSerializer(serializers.Serializer):  # pylint: disable=abstract-method
+    """Serializer for GenerateIntendedConfigView."""
+
+    intended_config = serializers.CharField()
+    intended_config_lines = serializers.ListField(child=serializers.CharField())

nautobot_golden_config/api/urls.py

@@ -17,11 +17,17 @@
 router.register("remediation-setting", views.RemediationSettingViewSet)
 router.register("config-postprocessing", views.ConfigToPushViewSet)
 router.register("config-plan", views.ConfigPlanViewSet)
-urlpatterns = router.urls
-urlpatterns.append(
+
+urlpatterns = [
     path(
         "sotagg/<uuid:pk>/",
         views.SOTAggDeviceDetailView.as_view(),
         name="device_detail",
-    )
-)
+    ),
+    path(
+        "generate-intended-config/",
+        views.GenerateIntendedConfigView.as_view(),
+        name="generate_intended_config",
+    ),
+]
+urlpatterns += router.urls