~/.jamal/.jim is included by the processor if it exists

documentation updated and some errors were fixed setting detectNoChange global
xml formatting writing to standard error was muted

FAQ.adoc

@@ -16,7 +16,8 @@ You do not need to know Java or be a Java programmer to use Jamal.
 
 Yes.
 You can use Jamal in your Gradle built project.
-There is no Gradle plugin or Gradle extension the same way as there are plugin and extension for Maven.
+There is a separate Gradle plugin project.
+There is  no Gradle extension the same way as there is an extension for Maven.
 The reason for that is, you do not need it.
 Gradle is configured using a powerful Groovy based DSL that can invoke any Java API, hence it does not need any Jamal like preprocessor.
 
@@ -44,11 +45,13 @@ Note that even when you changed the opening and closing strings if you
 
 then `{` and } will be used as macro opening and closing strings inside the included file.
 
-Also, when you use the IntelliJ Asciidoctor plugin with the Jamal preprocessor, the macro opening string is
+Also, when you use the IntelliJ Asciidoctor plugin with the Jamal preprocessor,
 
-* `{%`, and the macro closing string is
+* the macro opening string is `{%`, and
 
-* `%}`, unless the file starts with the characters  `{@`.
+* the macro closing string is `%}`,
+
+unless the file starts with the characters  `{@`.
 
 
 === I installed the ASCIIDOC plugin, and it gives error for the Jamal documentation

FAQ.adoc.jam

@@ -14,7 +14,8 @@ You do not need to know Java or be a Java programmer to use Jamal.
 {Q/Is there support for Gradle?/
 Yes.
 You can use Jamal in your Gradle built project.
-There is no Gradle plugin or Gradle extension the same way as there are plugin and extension for Maven.
+There is a separate Gradle plugin project.
+There is  no Gradle extension the same way as there is an extension for Maven.
 The reason for that is, you do not need it.
 Gradle is configured using a powerful Groovy based DSL that can invoke any Java API, hence it does not need any Jamal like preprocessor.
 }
@@ -40,13 +41,15 @@ Note that even when you changed the opening and closing strings if you
 
 then `{@escape*``{``}` and {@escape*``}``} will be used as macro opening and closing strings inside the included file.
 
-Also, when you use the IntelliJ Asciidoctor plugin with the Jamal preprocessor, the macro opening string is
+Also, when you use the IntelliJ Asciidoctor plugin with the Jamal preprocessor,
 
-* `{@escape*``{%``}`, and the macro closing string is
+* the macro opening string is `{@escape*``{%``}`, and
 
-* `{@escape*``%}``}`, unless the file starts with the characters  `{@escape*``{@``}`.
+* the macro closing string is `{@escape*``%}``}`,
 
+unless the file starts with the characters  `{@escape*``{@``}`.
 }
+
 {Q|I installed the ASCIIDOC plugin, and it gives error for the Jamal documentation|
 If you try to look at the git master HEAD version there is a possibility that the documentation has errors.
 Check on GitHub if the build is broken.

GLOSSARY.adoc

@@ -29,7 +29,7 @@ User-defined macro classes are instantiated many times and may contain a state.
 The instance of the class `javax0.jamal.engine.UserDefinedMacro` is created by the built-in macro `define`.
 Other built-in macros can create instances of different classes, so long as long they implement the `javax0.jamal.api.UserDefinedMacro` interface.
 
-Identifier:: In Jamal macro names are identifiers.
+Identifier:: In Jamal, macro names are identifiers.
 Identifiers in Jamal start with the letters `a-z` or `A-Z`, `$`, `:`, or underscore `_` and can contain any of these on consecutive positions. The second and other positions can also contain digits.
 Identifiers in Jamal start with the letters `a-z` or `A-Z`, `$`, `:`, or underscore `_` and can contain any of these on consecutive positions. The second and other positions can also contain digits.
 

GLOSSARY.adoc.jam

@@ -37,7 +37,7 @@ User-defined macro classes are instantiated many times and may contain a state.
 The instance of the class {%@java:class javax0.jamal.engine.UserDefinedMacro%} is created by the built-in macro `define`.
 Other built-in macros can create instances of different classes, so long as long they implement the {%UDM%} interface.
 
-Identifier:: In Jamal macro names are identifiers.
+Identifier:: In Jamal, macro names are identifiers.
 Identifiers in Jamal start with the letters `a-z` or `A-Z`, `$`, `:`, or underscore `_` and can contain any of these on consecutive positions. The second and other positions can also contain digits.
 Identifiers in Jamal start with the letters `a-z` or `A-Z`, `$`, `:`, or underscore `_` and can contain any of these on consecutive positions. The second and other positions can also contain digits.
 

README.adoc

@@ -222,7 +222,8 @@ The configuration values are searched for in this order.
 Different macros use different configuration keys.
 They are documented along with the macro documentation.
 
-The environment variables and their meanings are documented in their link:../documentation/ENVIRONMENT_VARIABLES.adoc[documentation].
+The environment variables and their meanings are documented in their
+link:./documentation/ENVIRONMENT_VARIABLES.adoc[documentation].
 
 
 == Features [[Features]]

README.adoc.jam

@@ -207,7 +207,8 @@ The configuration values are searched for in this order.
 Different macros use different configuration keys.
 They are documented along with the macro documentation.
 
-The environment variables and their meanings are documented in their link:../documentation/ENVIRONMENT_VARIABLES.adoc[documentation].
+The environment variables and their meanings are documented in their
+link:./documentation/ENVIRONMENT_VARIABLES.adoc[documentation].
 
 {%section /Features/Features%}
 

RELEASES.adoc

@@ -22,6 +22,9 @@
 * There is support to run Jamal from docker in the `jamal-docker` module.
 * The core macro `for` with the parop `evalist` allows you to miss the macro opening and closing strings when the list is nothing but an argument-less macro invocation.
 * The `jamal-prog` package contains a macro `decimal` to support `BigDecimal` calculation in BASIC code.
+* The `replace` and `replaceLines` macros in the snippet package result in error not only if the input was not changed, but also when some of the search/replace string-string or regular expression-string pairs made no effect on the input when the option `detectNoChange` is used.
+This helps to avoid situations when a snippet is heavily transformed to create documentation lines from source code and the source changes structurally and the transformation gets outdated.
+It is recommended to set the `detectNoChange` option to true globally at the start of the input file.w33
 
 == https://github.com/verhas/jamal/tree/2.4.0[2.4.0]
 

RELEASES.adoc.jam

@@ -22,6 +22,9 @@
 * There is support to run Jamal from docker in the `jamal-docker` module.
 * The core macro `for` with the parop `evalist` allows you to miss the macro opening and closing strings when the list is nothing but an argument-less macro invocation.
 * The `jamal-prog` package contains a macro `decimal` to support `BigDecimal` calculation in BASIC code.
+* The `replace` and `replaceLines` macros in the snippet package result in error not only if the input was not changed, but also when some of the search/replace string-string or regular expression-string pairs made no effect on the input when the option `detectNoChange` is used.
+This helps to avoid situations when a snippet is heavily transformed to create documentation lines from source code and the source changes structurally and the transformation gets outdated.
+It is recommended to set the `detectNoChange` option to true globally at the start of the input file.w33
 
 {%RELEASE IZMIR%}
 

documentation/ENVIRONMENT_VARIABLES.adoc

@@ -20,6 +20,7 @@ First, the system property is consulted, and the environment variable has only e
 The following sections describe the individual environment variables.
 
 
+
 ==== `JAMAL_CONNECT_TIMEOUT`
 This variable can define the connection timeout value for the web download in millisecond as unit.
 Jamal can download resources when the name of a file starts with the prefix `https://`.

documentation/ENVIRONMENT_VARIABLES.adoc.jam

@@ -17,9 +17,10 @@ For every environment variable, there is a corresponding system property.
 The name of the system property is the same as the environment variable lower-case converted and replacing the `_` to `.` characters.
 For example, for the environment variable {%JAMAL_CHECKSTATE_ENV%}, the system property is `jamal.checkstate`.
 
-    {%#replace /{%JAMAL_CHECKSTATE_ENV%}/`/%}  --> jamal.checkstate
+    {%#replace (detectNoChange=false)/{%JAMAL_CHECKSTATE_ENV%}/`/%}  --> jamal.checkstate
 
 First, the system property is consulted, and the environment variable has only effect when the system property is not defined.
 The following sections describe the individual environment variables.
 {%@import res:snippet.jim%}\
+{%@option ~detectNoChange%}
 {%docFrom {%@include [verbatim] ../jamal-api/src/main/java/javax0/jamal/api/EnvironmentVariables.java%}%}

documentation/MODULES.adoc.jam

@@ -42,7 +42,7 @@ __{input}__ is a module implementing an input handler for Jamal.
 Input handlers are loaded via the service loader and can load resource files, maven artifacts, files from JAR archives, etc.
 {@counter:define id=chapter_counter}
 {@define chapter($x,$y)==== {chapter_counter}. link:https://github.com/verhas/jamal/blob/master/jamal-$x/README.adoc[{@case:upper $x}] ^_{$y}_^
-{#define ! _{#replace/$x/-/_/}=.}{@comment makes chapter doubly define an error}
+{#define ! _{#replace(detectNoChange=false)/$x/-/_/}=.}{@comment makes chapter doubly define an error}
 }
 
 == Modules

documentation/PAROPS.adoc

@@ -1,5 +1,6 @@
 = Standard Built-In Parop Parsing
 
+
 Parops (parameter options) are usually used to give extra parameters/options to built-in macros.
 They are key/value pars at the start of the input of the built-in macro.
 The parsing of these parops is supported by Java classes that the built-in macros can use.

documentation/PAROPS.adoc.jam

@@ -1,5 +1,6 @@
 = Standard Built-In Parop Parsing
 {%@define methodFormat=`$name()`%}\
+{%@option ~detectNoChange%}
 
 Parops (parameter options) are usually used to give extra parameters/options to built-in macros.
 They are key/value pars at the start of the input of the built-in macro.

documentation/macros/eval.adoc.jam

@@ -27,7 +27,7 @@ You may need the explicit specification of `jamal` when the content of the macro
 
 {%#define evaluateLoopLimit={%#replace (regex) /{%@snip evaluateLoopLimit /(".*?")/%}/"//%}%}\
 {%@define evaluateLoopLimitAliases={%#replace (regex) /{%@snip evaluateLoopLimit /".*?"\s*,\s*(".*")/%}/"/`/(.*),\s/$1, and /%}%}\
-{%@define firstAlias={%#replace (regex) /{%@snip evaluateLoopLimit /".*?"\s*,\s*"(.*?)"/%}/"/`/%}%}\
+{%@define firstAlias={%#replace (regex detectNoChange=false) /{%@snip evaluateLoopLimit /".*?"\s*,\s*"(.*?)"/%}/"/`/%}%}\
 {%#assert:intEquals /{%#string:length {%#replace (regex) /{%evaluateLoopLimitAliases%}/[\w`\s]//%}%}/1/there are more than two aliases%}\
 
 If character following the keyword `eval` is `\*`, then the scripting type is `jamal` and the evaluation is repeated until all macros are resolved.

documentation/macros/macrodoc.jim

@@ -13,7 +13,7 @@ sinc{%@comment%}e version `since`
 
 
 this macro is to be used to reference other macros inside the documentation of the package
-{%@define $($name)=link:{%#file (format=$name){%#case:lower {%@replace /$name/:/_%}%}.adoc%}[`$name`]%}
+{%@define $($name)=link:{%#file (format=$name){%#case:lower {%@replace (detectNoChange=false) /$name/:/_%}%}.adoc%}[`$name`]%}
 
 
 

jamal-api/src/main/java/javax0/jamal/api/EnvironmentVariables.java

@@ -272,6 +272,9 @@ public static Properties getNewProperties() throws IOException {
         return new PropertiesSingleton().properties;
     }
 
+    public static String getConfigDir() throws IOException {
+        return PropertiesSingleton.jamalDirectory;
+    }
     public static final String INSECURE_CONFIGURATION = "INSECURE CONFIGURATION: ";
 
     private static void assertPathPrivate(Path path) throws IOException {

jamal-asciidoc/README.adoc

@@ -24,7 +24,7 @@ Maintaining a separate `.adoc.jam` and the generated `.adoc` file is a good solu
 For example, when you maintain your project on GitHub, the `.adoc` file is rendered by the platform on the file.
 You can only do it if you have the Jamal preprocessed and generated `.adoc` file uploaded to GitHub.
 
-== Installation with IntelliJ IDEA
+== Installation with IntelliJ IDEA (and with other JetBrains IDEs)
 
 [NOTE]
 ====
@@ -34,6 +34,10 @@ It is an extension to Asciidoctor, which itself is an IntelliJ plugin.
 This way, this extension is an extension of a plugin.
 You cannot install this extension as a separate plugin to IntelliJ.
 It works only with the Asciidoc plugin in IntelliJ.
+
+A separate IntelliJ plugin for Jamal supporting code completion, documentation display for hovering is developed in an independent project by other developers.
+
+This preprocessor works for any JetBrains IDE that supports the Asciidoctor plugin.
 ====
 
 

jamal-asciidoc/README.adoc.jam

@@ -26,7 +26,7 @@ Maintaining a separate `.adoc.jam` and the generated `.adoc` file is a good solu
 For example, when you maintain your project on GitHub, the `.adoc` file is rendered by the platform on the file.
 You can only do it if you have the Jamal preprocessed and generated `.adoc` file uploaded to GitHub.
 
-== Installation with IntelliJ IDEA
+== Installation with IntelliJ IDEA (and with other JetBrains IDEs)
 
 [NOTE]
 ====
@@ -36,6 +36,10 @@ It is an extension to Asciidoctor, which itself is an IntelliJ plugin.
 This way, this extension is an extension of a plugin.
 You cannot install this extension as a separate plugin to IntelliJ.
 It works only with the Asciidoc plugin in IntelliJ.
+
+A separate IntelliJ plugin for Jamal supporting code completion, documentation display for hovering is developed in an independent project by other developers.
+
+This preprocessor works for any JetBrains IDE that supports the Asciidoctor plugin.
 ====
 {%#snip:define LAST_RELEASE=
 {%#killLines pattern=".*LAST_RELEASE.*" keep

jamal-asciidoc/install-asciidoc.sh

@@ -17,4 +17,4 @@ unzip ~/.m2/repository/com/javax0/jamal/jamal-asciidoc/2.5.0-SNAPSHOT/jamal-asci
 echo "cd back to jamal-asciidoc project directory"
 popd
 echo "Restart IntelliJ, then you have the new version"
-echo "DONE"
+echo "DONE"

jamal-cmd/src/test/resources/jamal.sh.template.jam

@@ -32,7 +32,7 @@ The following lines include the file dependencies.txt created by the dependency
 and stra
 }
 
-{#replaceLines replace="/^\\s*(.*?):(.*?):jar:(.*?):compile.*$/download \"$1\" \"$2\" \"$3\"" replace="/^\\s*//"
+{#replaceLines replace="/^\\s*(.*?):(.*?):jar:(.*?):compile.*$/download \"$1\" \"$2\" \"$3\""
   {#killLines keep pattern=compile
     {#include [verbatim] {ROOT}/jamal-cmd/target/dependencies.txt}}}
 

jamal-engine/src/main/java/javax0/jamal/engine/Processor.java

@@ -25,7 +25,9 @@
 import javax0.jamal.tracer.TraceRecord;
 import javax0.jamal.tracer.TraceRecordFactory;
 
+import java.io.File;
 import java.io.IOException;
+import java.io.InputStream;
 import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayDeque;
@@ -110,12 +112,17 @@ public Processor(String macroOpen, String macroClose, Context context) {
         URL url = null;
         try {
             macros.separators("{", "}");
+            final var globalIncludeFile = new File(EnvironmentVariables.getConfigDir() + "/" + GLOBAL_INCLUDE_RESOURCE);
+            if (globalIncludeFile.exists()) {
+                try (final var is = globalIncludeFile.toURI().toURL().openStream()) {
+                    processInputStream(is);
+                }
+            }
             final var urls = getClass().getClassLoader().getResources(GLOBAL_INCLUDE_RESOURCE);
             while (urls.hasMoreElements()) {
                 url = urls.nextElement();
                 try (final var is = url.openStream()) {
-                    final var in = makeInput(new String(is.readAllBytes(), StandardCharsets.UTF_8), new Position("res:" + GLOBAL_INCLUDE_RESOURCE, 1, 1));
-                    process(in);
+                    processInputStream(is);
                 }
             }
             macros.separators(macroOpen, macroClose);
@@ -144,6 +151,11 @@ public Processor() {
         this("{", "}");
     }
 
+    private void processInputStream(InputStream is) throws IOException, BadSyntax {
+        final var in = makeInput(new String(is.readAllBytes(), StandardCharsets.UTF_8), new Position("res:" + GLOBAL_INCLUDE_RESOURCE, 1, 1));
+        process(in);
+    }
+
     @Override
     public UserDefinedMacro newUserDefinedMacro(String id, String input, String... params) throws BadSyntax {
         return newUserDefinedMacro(id, input, false, false, params);
@@ -165,7 +177,7 @@ public ScriptMacro newScriptMacro(String id, String scriptType, String input, St
     }
 
     @Override
-    public Processor spawn(){
+    public Processor spawn() {
         return new Processor(macros.open(), macros.close(), context);
     }
 
@@ -234,7 +246,7 @@ public JShellEngine getJShellEngine() {
     /**
      * Process the text at the start of input till the first macro start.
      *
-     * @param input  where the text is read from and removed afterwards
+     * @param input  where the text is read from and removed afterward
      * @param output where the text is appended
      */
     private void processText(Input input, Input output) {
@@ -513,7 +525,7 @@ private String evalUserDefinedMacro(final Input input, final TraceRecord tr, fin
         qualifier.macroId = id;
         skipWhiteSpaces(evaluatedInput);
         final Optional<Identified> identifiedOpt;
-        if (id.length() == 0) {
+        if (id.isEmpty()) {
             identifiedOpt = Optional.of(new NullMacro(qualifier.processor));
         } else {
             if (reportUndef || !optionsStore.is(NO_UNDEFAULT)) {

jamal-io/README.adoc.jam

@@ -260,13 +260,14 @@ They all have to be specified in the first line of the macro.%}{%ALIAS$ONLY%}
 
 {%@define example($x)=[source]
 ----
-{%#replaceLines replace="~System\\.setProperty\\(\"(\\w+)\", \"(\\w+)\"\\);~$1 -> $2~" \
-                replace=~^\s*~~ \
-                replace="~([^\\\\])\"~$1~" \
-                replace="~^\"~~" \
-                replace="~\\+$~~" \
-                replace="~\\\\\"~\"~" \
-                replace="~\\\\n~\\n~"
+{%#replaceLines detectNoChange=false \
+                replace="~System\\.setProperty\\(\"(\\w+)\", \"(\\w+)\"\\);~$1 -> $2~" \
+                replace=/^\s* \
+                replace="/([^\\\\])\"/$1" \
+                replace="/^\"" \
+                replace=/\+$ \
+                replace=/\\"/\" \
+                replace=/\\n/\n
 {%@snip $x%}
 %}
 ----%}
@@ -421,7 +422,10 @@ In this case the output is typically redirected to a file and the result can be
 The macro `waitFor` uses a subset of the options of the `exec` macro.{%ALIAS$ONLY%}
 
 {%#eval {%`@comment%}
-{%#replaceLines replace="~.*?final\\s+var\\s+\\w+\\s*=\\s*.*?\\(null,(.*?)\\).*~* $1~.*//\\s*~" replace="~\"(\\w)~`$1~(\\w)\"~$1`~"
+{%#replaceLines replace="~.*?final\\s+var\\s+\\w+\\s*=\\s*.*?\\(null,(.*?)\\).*~* $1" \
+                replace="~.*//\\s*~" \
+                replace="~\"(\\w)~`$1"\
+                replace="~(\\w)\"~$1`~"
 {%@snip waitFor_options%}
 %}%}