Add support for realm normalization

.gitignore

@@ -19,3 +19,4 @@ release.properties
 
 /*.json
 /test*
+/exports

docs/FEATURES.md

@@ -57,6 +57,7 @@
 | Synchronize user profile                           | 5.4.0 | Synchronize the user profile configuration defined on the realm configuration                            |
 | Synchronize client-policies                        | 5.6.0 | Synchronize the client-policies (clientProfiles and clientPolicies) while updating realms                |
 | Synchronize message bundles                        | 5.12.0 | Synchronize message bundles defined on the realm configuration                                           |
+| Normalize realm exports                            | x.x.x | Normalize a full realm export to be more minimal                                                         |
 
 # Specificities
 

docs/NORMALIZE.md

@@ -0,0 +1,123 @@
+# Realm normalization
+
+Realm normalization is a feature that is supposed to aid users in migrating from an "unmanaged" Keycloak installation,
+to an installation managed by keycloak-config-cli.
+To achieve this, it uses a full [realm export](https://www.keycloak.org/server/importExport#_exporting_a_specific_realm)
+as an input, and only retains things that deviate from the default
+
+## Usage
+
+To run the normalization, run keycloak-config-cli with the CLI option `--run.operation=NORMALIZE`.
+The default value for this option is `IMPORT`, which will run the regular keycloak-config-cli import.
+
+### Configuration options
+
+| Configuration key                    | Purpose                                                                                                         | Example       |
+|--------------------------------------|-----------------------------------------------------------------------------------------------------------------|---------------|
+| run.operation                        | Tell keycloak-config-cli to normalize, rather than import                                                       | NORMALIZE     |
+| normalization.files.input-locations  | Which realm files to import                                                                                     | See IMPORT.md |
+| normalization.files.output-directory | Where to save output realm files                                                                                | ./exports/out |
+| normalization.output-format          | Whether to output JSON or YAML. Default value is YAML                                                           | YAML          |
+| normalization.fallback-version       | Use this version as a baseline of keycloak version in realm is not available as baseline in keycloak-config-cli | 19.0.3        |
+
+### Unimplemented Features
+- Components:
+  - Currently, keycloak-config-cli will not yet look at the `components` section of the exported JSON
+  - Therefore, some things (like LDAP federation configs and Key providers) are missing from the normalized YAML
+- Users
+  - Users are not currently considered by normalization.
+
+## Missing entries
+keycloak-config-cli will WARN if components that are present in a realm by default are missing from an exported realm.
+An example of such a message is:
+```
+Default realm requiredAction 'webauthn-register-passwordless' was deleted in exported realm. It may be reintroduced during import
+```
+Messages like these will often show up when using keycloak-config-cli to normalize an import from an older version of Keycloak, and compared to a newer baseline.
+In the above case, the Keycloak version is 18.0.3, and the baseline for comparison was 19.0.3.
+Since the webauthn feature was not present (or enabled by default) in the older version, this message is mostly informative.
+If a message like this appears on a component that was *not* deleted or should generally be present, this may indicate a bug in keycloak-config-cli.
+
+## Cleaning of invalid data
+Sometimes, realms of existing installations may contain invalid data, due to faulty migrations, or due to direct interaction with the database,
+rather than the Keycloak API.
+When such problems are found, we attempt to handle them in keycloak-config-cli, or at least notify the user about the existence of these problems.
+
+### SAML Attributes on clients
+While not necessarily invalid, openid-connect clients that were created on older Keycloak versions, will sometimes contain
+SAML-related attributes. These are filtered out by keycloak-config-cli.
+
+### Unused non-top-level Authentication Flows
+Authentication flows in Keycloak are marked as top-level if they are supposed to be available for binding or overrides.
+Authentication flows that are not marked as top-level are used as sub-flows in other authentication flows.
+The normalization process recognizes recursively whether there are authentication flows that are not top level and not used
+by any top level flow, and does not include them in the final result.
+
+A warning message is logged, and you can use the following SQL query to find any authentication flows that are not referenced.
+Note that this query, unlike keycloak-config-cli, is not recursive.
+That means that after deleting an unused flow, additional unused flows may appear after the query is performed again.
+
+```sql
+select flow.alias
+from authentication_flow flow
+         join realm r on flow.realm_id = r.id
+         left join authentication_execution execution on flow.id = execution.auth_flow_id
+where r.name = 'mytest'
+  and execution.id is null
+  and not flow.top_level
+```
+
+### Unused and duplicate Authenticator Configs
+Authenticator Configs are not useful if they are not referenced by at least one authentication execution.
+Therefore, keycloak-config-cli detects unused configurations and does not include them in the resulting output.
+Note that the check for unused configs runs *after* the check for unused flows.
+That means a config will be detected as unused if it is referenced by an execution that is part of a flow that is unused.
+
+A warning message is logged on duplicate or unused configs, and you can use the following SQL query to find any configs
+that are unused:
+
+```sql
+select ac.alias, ac.id
+from authenticator_config ac
+         left join authentication_execution ae on ac.id = ae.auth_config
+         left join authentication_flow af on ae.flow_id = af.id
+         join realm r on ac.realm_id = r.id
+where r.name = 'master' and af.alias is null
+order by ac.alias
+```
+
+And the following query to find duplicates:
+
+```sql
+select alias, count(alias), r.name as realm_name
+from authenticator_config
+         join realm r on realm_id = r.id
+group by alias, r.name
+having count(alias) > 1
+```
+
+If the `af.id` and `af.alias` fields are `null`, the config in question is not in use.
+Note that configs used by unused flows are not marked as unused in the SQL result, as these need to be deleted first
+to become unused.
+After the unused flows (and executions) are deleted, the configs will be marked as unused and can also be deleted.
+
+### Authentication Executions with invalid subflows
+Some keycloak exports have invalid authentication executions that reference a subflow, while also setting an authenticator.
+This is only a valid configuration if the subflow's type is `form-flow`.
+If it is not, then keycloak-config-cli will not import the configuration.
+This will be marked by an ERROR severity message in the log output.
+You can use this SQL query to find offending entries and remediate the configuration errors before continuing.
+
+```sql
+select parent.alias,
+       subflow.alias,
+       execution.alias
+from authentication_execution execution
+         join realm r on execution.realm_id = r.id
+         join authentication_flow parent on execution.flow_id = parent.id
+         join authentication_flow subflow on execution.auth_flow_id = subflow.id
+where execution.auth_flow_id is not null
+  and execution.authenticator is not null
+  and subflow.provider_id <> 'form-flow'
+  and r.name = 'REALMNAME';
+```

pom.xml

@@ -74,6 +74,7 @@
         <keepachangelog.version>2.1.1</keepachangelog.version>
         <license-plugin.version>2.4.0</license-plugin.version>
         <logstash-logback-encoder.version>8.0</logstash-logback-encoder.version>
+        <javers.version>7.6.3</javers.version>
         <maven-failsafe-plugin.version>3.2.5</maven-failsafe-plugin.version>
         <maven-release-plugin.version>3.1.1</maven-release-plugin.version>
         <maven-replacer.version>1.5.3</maven-replacer.version>
@@ -129,6 +130,30 @@
                 <scope>import</scope>
                 <type>pom</type>
             </dependency>
+
+            <dependency>
+                <groupId>org.apache.commons</groupId>
+                <artifactId>commons-text</artifactId>
+                <version>${commons-text.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.commons</groupId>
+                <artifactId>commons-lang3</artifactId>
+                <version>${commons-lang3.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>net.jodah</groupId>
+                <artifactId>failsafe</artifactId>
+                <version>${failsafe.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.javers</groupId>
+                <artifactId>javers-core</artifactId>
+                <version>${javers.version}</version>
+            </dependency>
         </dependencies>
     </dependencyManagement>
 
@@ -170,12 +195,6 @@
             <artifactId>jackson-datatype-jdk8</artifactId>
         </dependency>
 
-        <dependency>
-            <groupId>org.yaml</groupId>
-            <artifactId>snakeyaml</artifactId>
-            <version>${snakeyaml.version}</version>
-        </dependency>
-
         <dependency>
             <groupId>org.keycloak</groupId>
             <artifactId>keycloak-admin-client</artifactId>
@@ -209,6 +228,26 @@
             <version>${failsafe.version}</version>
         </dependency>
 
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>com.fasterxml.jackson.dataformat</groupId>
+            <artifactId>jackson-dataformat-yaml</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.javers</groupId>
+            <artifactId>javers-core</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.yaml</groupId>
+            <artifactId>snakeyaml</artifactId>
+        </dependency>
+
         <!-- JSON logging -->
         <dependency>
             <groupId>net.logstash.logback</groupId>

src/main/java/de/adorsys/keycloak/config/KeycloakConfigApplication.java

@@ -20,14 +20,13 @@
 
 package de.adorsys.keycloak.config;
 
-import de.adorsys.keycloak.config.properties.ImportConfigProperties;
-import de.adorsys.keycloak.config.properties.KeycloakConfigProperties;
+import de.adorsys.keycloak.config.properties.RunConfigProperties;
 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
 import org.springframework.boot.context.properties.EnableConfigurationProperties;
 
 @SpringBootApplication(proxyBeanMethods = false)
-@EnableConfigurationProperties({KeycloakConfigProperties.class, ImportConfigProperties.class})
+@EnableConfigurationProperties(RunConfigProperties.class)
 public class KeycloakConfigApplication {
     public static void main(String[] args) {
         // https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-application-exit

src/main/java/de/adorsys/keycloak/config/KeycloakConfigNormalizationRunner.java

@@ -0,0 +1,126 @@
+/*-
+ * ---license-start
+ * keycloak-config-cli
+ * ---
+ * Copyright (C) 2017 - 2022 adorsys GmbH & Co. KG @ https://adorsys.com
+ * ---
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * ---license-end
+ */
+
+package de.adorsys.keycloak.config;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLMapper;
+import de.adorsys.keycloak.config.properties.NormalizationConfigProperties;
+import de.adorsys.keycloak.config.properties.NormalizationKeycloakConfigProperties;
+import de.adorsys.keycloak.config.provider.KeycloakExportProvider;
+import de.adorsys.keycloak.config.service.normalize.RealmNormalizationService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.ExitCodeGenerator;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.stereotype.Component;
+
+import java.io.FileOutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+import static de.adorsys.keycloak.config.properties.NormalizationConfigProperties.OutputFormat.YAML;
+
+@Component
+@ConditionalOnProperty(prefix = "run", name = "operation", havingValue = "NORMALIZE")
+@EnableConfigurationProperties({NormalizationConfigProperties.class, NormalizationKeycloakConfigProperties.class})
+public class KeycloakConfigNormalizationRunner implements CommandLineRunner, ExitCodeGenerator {
+
+    private static final Logger logger = LoggerFactory.getLogger(KeycloakConfigNormalizationRunner.class);
+    private static final long START_TIME = System.currentTimeMillis();
+
+    private final RealmNormalizationService normalizationService;
+    private final KeycloakExportProvider exportProvider;
+    private final NormalizationConfigProperties normalizationConfigProperties;
+    private final YAMLMapper yamlMapper;
+    private final ObjectMapper objectMapper;
+    private int exitCode;
+
+    @Autowired
+    public KeycloakConfigNormalizationRunner(RealmNormalizationService normalizationService,
+                                             KeycloakExportProvider exportProvider,
+                                             NormalizationConfigProperties normalizationConfigProperties,
+                                             YAMLMapper yamlMapper,
+                                             ObjectMapper objectMapper) {
+        this.normalizationService = normalizationService;
+        this.exportProvider = exportProvider;
+        this.normalizationConfigProperties = normalizationConfigProperties;
+        this.yamlMapper = yamlMapper;
+        this.objectMapper = objectMapper;
+    }
+
+    @Override
+    public void run(String... args) throws Exception {
+        try {
+            var outputLocation = Paths.get(normalizationConfigProperties.getFiles().getOutputDirectory());
+            if (!Files.exists(outputLocation)) {
+                logger.info("Creating output directory '{}'", outputLocation);
+                Files.createDirectories(outputLocation);
+            }
+            if (!Files.isDirectory(outputLocation)) {
+                logger.error("Output location '{}' is not a directory. Aborting", outputLocation);
+                exitCode = 1;
+                return;
+            }
+
+            for (var exportLocations : exportProvider.readFromLocations().values()) {
+                for (var export : exportLocations.entrySet()) {
+                    logger.info("Normalizing file '{}'", export.getKey());
+                    for (var realm : export.getValue()) {
+                        var normalizedRealm = normalizationService.normalizeRealm(realm);
+                        var suffix = normalizationConfigProperties.getOutputFormat() == YAML ? "yaml" : "json";
+                        var outputFile = outputLocation.resolve(String.format("%s.%s", normalizedRealm.getRealm(), suffix));
+                        try (var os = new FileOutputStream(outputFile.toFile())) {
+                            if (normalizationConfigProperties.getOutputFormat() == YAML) {
+                                yamlMapper.writeValue(os, normalizedRealm);
+                            } else {
+                                objectMapper.writeValue(os, normalizedRealm);
+                            }
+                        }
+                    }
+                }
+            }
+        } catch (NullPointerException e) {
+            throw e;
+        } catch (Exception e) {
+            logger.error(e.getMessage());
+
+            exitCode = 1;
+
+            if (logger.isDebugEnabled()) {
+                throw e;
+            }
+        } finally {
+            long totalTime = System.currentTimeMillis() - START_TIME;
+            String formattedTime = new SimpleDateFormat("mm:ss.SSS").format(new Date(totalTime));
+            logger.info("keycloak-config-cli running in {}.", formattedTime);
+        }
+    }
+
+    @Override
+    public int getExitCode() {
+        return exitCode;
+    }
+}

src/main/java/de/adorsys/keycloak/config/KeycloakConfigRunner.java

@@ -23,13 +23,16 @@
 import de.adorsys.keycloak.config.model.KeycloakImport;
 import de.adorsys.keycloak.config.model.RealmImport;
 import de.adorsys.keycloak.config.properties.ImportConfigProperties;
+import de.adorsys.keycloak.config.properties.KeycloakConfigProperties;
 import de.adorsys.keycloak.config.provider.KeycloakImportProvider;
 import de.adorsys.keycloak.config.service.RealmImportService;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.CommandLineRunner;
 import org.springframework.boot.ExitCodeGenerator;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
 import org.springframework.stereotype.Component;
 
 import java.text.SimpleDateFormat;
@@ -39,6 +42,13 @@
 import java.util.Map;
 
 @Component
+/*
+ * Spring only considers actual properties set, not default values of @ConfigurationProperties classes.
+ * Therefore, we enable matchIfMissing here, so if there is *no* property set, we consider it an import
+ * for backwards compatibility
+ */
+@ConditionalOnProperty(prefix = "run", name = "operation", havingValue = "IMPORT", matchIfMissing = true)
+@EnableConfigurationProperties({ImportConfigProperties.class, KeycloakConfigProperties.class})
 public class KeycloakConfigRunner implements CommandLineRunner, ExitCodeGenerator {
     private static final Logger logger = LoggerFactory.getLogger(KeycloakConfigRunner.class);
     private static final long START_TIME = System.currentTimeMillis();