Skip to content

Latest commit

 

History

History
642 lines (457 loc) · 24.1 KB

README.adoc

File metadata and controls

642 lines (457 loc) · 24.1 KB
Raw

Jamal IO module

Using this module, you can handle files from Jamal source code. You can

  • create a file,

  • write text to the file,

  • create directories,

  • delete directories, files and

  • you can print messages to the standard output or to the standard error.

To use this module, you have to add the dependency to your Maven project, as:

<dependency>
    <groupId>com.javax0.jamal</groupId>
    <artifactId>jamal-io</artifactId>
    <version>2.8.2-SNAPSHOT</version>
</dependency>

or as an alternative your macro file can use the maven:load macro to load the module as

Jamal source
{@maven:load com.javax0.jamal:jamal-io:2.8.2-SNAPSHOT}

before using any macro from this package for the first time. Also note, that using the maven:load macro requires that you configure the jamal-io package as safe in the ~/.jamal/settings.properties or ~/.jamal/settings.xml file.

Following that, you can use the macros

macros.

Macros implemented in the package

i. io:cwd

This macro can be used to get the current working directory. The format of the macro is

Jamal source
{@io:cwd}

and the output is the actual working directory absolute path name.

ii. os:name

This macro will return the actual operating system name, as it is contained in the system variable os.name.

iii. io:write

Using this macro, you can write some text into a file. The format of the macro is

Jamal source
{@io:write (options) text to write into the file}

The macro will write the input of the macro to a text file. The name of the file and write mode can be controlled using options. The options are:

  • io:outputFile (aliases io:output, output, io:file, file) can define the name of the file.

Use the user defined macro if you want to write several different segments to the file, and you do not want to specify the name of the file every the time.

  • io:append (alias append) is a boolean option controlling if the content should be appended, or the file has to be truncated. The default value is to truncate the file.

  • io:mkdir (alias mkdir) is a boolean option controlling if the directory where the file has to be is created before writing the file if it does not exist. The default is not to create the directory and hence, neither the file.

Note that the option can be defined as a user-defined macro or as an option using the {@option …​} macro. The aliases can only be used on the macro itself after the name of the macro between ( and ) characters.

iv. io:copy

This macro can be used to copy a file. It is usually useful when a source file is some binary information via the web, and you want to store it locally attached to the document. For example, the OpenAI can ask the service to generate a picture. The result is a downloadable picture. The picture is stored by the service only for a limited time. After that time, the picture us deleted and not available for the document. Using this macro, the picture can be downloaded and stored locally.

The macro is controlled by options. The options are:

  • from should specify the source file. It is usually a URL starting with https:. Note that Jamal deliberately does not support http: URLs.

  • to the target file name. This is where the file will be saved.

  • io:append (alias append) is a boolean parameter meaning that the file should be appended.

  • io:mkdir (alias mkdir) is a boolean parameter meaning that the directory where the file should be saved should be created if it does not exist.

  • cache is a boolean parameter meaning that the file download cache implemented in Jamal should be used.

  • overwrite is a boolean parameter meaning that the file should be overwritten if it exists. If this parameter is not specified and the target file exists, then the macro will not even start the copy. The same effect could be reached surrounding the io:copy macro with an if macro using {%link file%} checking the existence of the target file. This option is for convenience.

v. io:file

This macro can be used to test a file. The result of the macro will be either true or false depending on the file test.

The macro has two options:

  • io:file (alias file) names the file. The file name can be absolute or relative to the file where the macro is used.

  • isHidden, exists, isDirectory, isFile, canExecute, canRead, canWrite specifies the test. Technically, this option is boolean, but the macro tests the name you use instead of the existence. If you do not specify any of these tests then exists will be used.

vi. io:remove

This macro can be used to remove a file or directory. The format of the macro is

Jamal source
{@io:remove options}

The options are:

  • io:outputFile (aliases io:output, output, io:file, file) can define the name of the file.

  • io:recursive (alias recursive) is a boolean option controlling if the deletion should be recursive

Note that this macro reads the options directly from the input, and they are not enclosed between ( and ) characters.

Note that the option can be defined as a user-defined macro or as an option using the {@option …​} macro. The aliases can only be used on the macro itself after the name of the macro between ( and ) characters.

vii. io:mkdir

This macro can be used to create a directory. The format of the macro is

Jamal source
{@io:mkdir options}

The options are:

  • io:outputFile (aliases io:output, output, io:file, file) can define the name of the file.

  • io:recursive (alias recursive) is a boolean option controlling if the deletion should be recursive

Note that this macro reads the options directly from the input, and they are not enclosed between ( and ) characters.

Note that the option can be defined as a user-defined macro or as an option using the {@option …​} macro. The aliases can only be used on the macro itself after the name of the macro between ( and ) characters.

viii. io:print

This macro can be used to print some text to the standard output or to the standard error. The format of the macro is

Jamal source
{@io:print (options) message to print}

There is one option.

  • io:err (alias err) is a boolean option controlling if the message should be written to the standard output or to the standard error. The default is the standard output.

Note that the option can be defined as a user-defined macro or as an option using the {@option …​} macro. The aliases can only be used on the macro itself after the name of the macro between ( and ) characters.

ix. io:exec

This macro can start an external program. The typical use is to start an external document handling program, like Graphviz, which cannot be integrated in-process. The format of the macro is

{@io:exec options
input text}

The first line of the macro following the name of the macro contains the options. The rest of the macro will be used as the input text to the program, and Jamal will feed it into the standard input of the program.

Note that it is not possible to execute an arbitrary program from Jamal. Anything you want to execute as a separate process has to be configured in the system. For security reason, the command specification is a symbolic name. The executable should be configured in an environment variable, a system property or a Jamal configuration in the ~/.jamal/settings.properties or ~/.jamal/settings.xml file. The recommended way to configure the executable is to use the ~/.jamal/settings.properties or ~/.jamal/settings.xml file.

For example, if you want to execute the Graphviz program, you can configure it in the ~/.jamal/settings.properties file as:

Graphviz=/usr/local/bin/dot

After this you can execute the macro

{@io.exec command=Graphviz}

This will start the program without any argument, defined timeout or input text.

The options of the macro are defined as follows:

  • osOnly, os defines a pattern for the operating system’s name. The execution will only start if the operating system’s name matches the pattern. The pattern is a regular expression. The pattern is matched against the operating system’s name using the Java pattern matching find() method. It means that it is enough to provide a pattern that matches part of the OS name. For example, windows will match Windows 10 and Windows 7 but not Linux. If the pattern is not provided, the execution will start on all operating systems.

  • input defines the file name to be used as standard input for the new process. If it is not provided, then the content of the macro will be used as input. When an input is defined, the content of the macro will be ignored.

  • output defines the file name to be used as standard output for the new process. If it is not provided, then the output will appear as the result of the macro. When an output is defined, the result of the macro will be empty string.

  • error defines the file name to be used as standard error for the new process. If it is not provided, then the standard error will be used.

  • command The name of the command to be executed. This is not the name of the shell script or any executable. For security reason, every executable should be configured via a system property, environment variable or in the ~/.jamal/settings.properties file. The command itself is the string value of the configuration property. The search for the variables first looks at the system properties, then the environment variables and finally in the settings file. The name for these is converted to follow the system property and environment variable conventions. It means that the name MERMAID will be searched as mermaid when looking in the configuration file or as a system property. (MERMAID is an example, replace it with any name.) Also underscore and dot characters are converted back and forth.

    To ease typing, this parameter can be multi-line strings. In that case, the non-empty lines are treated as individual parameters before any arguments parameters are added. Must not start with empty line. The first line has to be the configured name of the command.

  • argument, arguments The arguments to be passed to the command. This is a multivalued parameter. To ease typing, each parameter can be multi-line strings. In that case, the non-empty lines are treated as individual parameters.

  • environment, env This option can specify the environment variables to be passed to the command. This option usually is a multi-line string, thus the use of the """ delimiter is recommended. Each line of the configuration parameter can be

    • empty, in which case the line is ignored

    • a comment starting with the # character, in which case the line is ignored

    • a key=value pair, in which case the key is the name of the environment variable and the value is the value of the variable.

    These variables are available for the command, but not for the Jamal process. You cannot use this parameter to define the environment variable specifying the executable. It would be convenient, but at the same time, it would just wipe out all the security measures introduced with the configuration requirements.

  • envReset, reset This option can be used to reset the environment variables before the command is executed. Without these options, the command will inherit the environment variables of the Jamal process, and the defined environment variables are added to the current list.

  • directory, cwd, curdir, cd Set the current working directory for the command. If this option is not provided, the current working directory of the Jamal process will be used.

  • async, asynch, asynchronous Using this option, Jamal will not wait for the command to finish before continuing with the next macro. In this case, the output cannot be used as the result of the macro. If this option is used, the output of the macro will be empty string. The value of this option has to be a macro name, which will be defined and will hold the reference to the process. This macro can later be used to wait for the process to finish. Although technically the name is a user defined macro, you cannot use it as a conventional user defined macro. It does not have any value and whenever the code would evaluate the macro it will result an error. Similarly, the name MUST NOT be defined as a user defined macro at the time the exec macro is evaluated. The exec macro handles the name as the core built-in macro define when a ! is used after the macro name. If there is a user defined macro of the same name on the same level, an error will occur.

  • wait, waitMax, timeOut This option can be used to specify the maximum amount of time in milliseconds to wait for the process to finish. If the process does not finish in the specified time, a BadSyntax exception will be thrown. This option cannot be used together with the async option.

  • destroy, kill This option can be used to destroy the process if it has not finished within the specified time. This option can only be used together with the wait option.

  • force, forced This option instructs the macro to destroy the process forcibly. This option can only be used together with the destroy option.

  • optional This option tells the macro to skip the execution of the command is not configured. If the macro uses the option asynch the process id will still be defined without a process. Any io:waitFor macro waiting for this process should also use the optional option.

Note that all these options are technically aliases. It means that you cannot use a user defined macro to specify their values. They all have to be specified in the first line of the macro.

Examples

In the followings we will list some examples of the use of the macro exec. These examples are collected from the integration test file src/test/java/javax0/jamal/io/TestExec.java. The first line of the examples is the definition of the command in the format symbol → value. The integration test sets these values as Java system properties. The rest of the lines contain the macro as it appears in the test code.

Note

When Jamal looks for some configuration it looks at the

  • system properties

  • environment variables

  • ~/.jamal/settings.(properties|xml)

whichever it finds first. The key given is used as is in the case of the environment variables. For example, JAVA_HOME is used as is. However, when the code looks at the system properties, it looks for the key java.home. The transformation is to contert to lower case and replace the underscore characters with dot. In the configuration file the key is also lowe case and the underscore characters are replaced with dot but if the key has a jamal. prefix it is also removed. This is the reason why the sample code defines exec in lower case and EXEC in upper case in the macro.

This example starts java to echo the version of the installed and used Java.

exec -> java
{@io:exec command=EXEC argument="-version"}

This example will print the current working directory. Because the current working directory is changed by the option cwd=target the result will be this directory. Note, however, that changing the working directory for the new process does not effect the parameters of the macro. The other parameters, like output still have to define the file names absolute, or relative to the file containing the macro.

exec -> pwd
{@io:exec command=EXEC cwd=target output="target/hallo.txt"}
{@include [verbatim] target/hallo.txt}

The following example calls the command cat which copies the standard input to the standard output. The standard input is not defined in the macro, therefore the text after the first line is used. The output is redirected into a file. The file will contain the text from the macro.

exec -> cat
{@io:exec command=EXEC output="target/catoutput.txt"n
hello, this is the text for the file}
Note

This is a system dependent and rather slow way to write something into a file. The io module provides a more efficient way to write into a file.

The next example calls the echo program that prints the argument to the standard output. Since no output file is defined the output is the result of the macro.

exec -> echo
{@io:exec command=EXEC argument="hello"}

The next sample calls a shell script. The content of the schellscript is

sleep 1
echo hello

The command is invoked asynchronously. It means that the macro does not wait for the completion of the process. The output of the process is not redirected to a file, and because it is asynchronous the output is thrown away. The result of the macro is empty string. The option async defines a name for the process, PROC001. This name can later be used to reference the process in the macro waitFor. In this example we do not wait for the process to finish, not even later.

exec -> sh
{@io:exec asynch=PROC001 command=EXEC argument=target/async.sh}

The next example calls the sleep program that sleeps for 1000 of seconds. We start the process in a synchronous mode and we wait for it 1000 milliseconds. Note that the argument to the proces, sleep is 1000 and the timeout value is also 1000. However, the program sleep interprets the argument in seconds, while the option wait is milliseconds. Evidently the wait time will timeout and after that Jamal will stop the external process.

exec -> sleep
{@io:exec command=EXEC argument=1000 wait=1000 destroy}

This example is a demo setting the environment variables. The external program prints out the environment variable AAA. The macro sets the environment variable AAA to BBB. The example shows a multipline example of environment variable setting demonstrating empty line and a comment line as well. The new value is added to the existing envrionment variables that the new process inherits from the Jamal executing process.

exec -> printenv
{@io:exec command=EXEC argument=AAA env="AAA=BABA\n\n #  oooh my\n"}

This exaple is similar to the previous one,but it resets the environment variables. The environment printout in the new process will print the value of the environment varianle JAVA_HOME. This environment variable should be defined in the environment where Jamal runs because Jamal is written in Java. On the other hand the external program will see this environment variable as undefined and the output of printenv is an empty string.

exec -> printenv
{@io:exec command=EXEC argument=JAVA_HOME envReset env="AAA=BABA"}

The next example shows how to use the option optional. This option tells the macro exec not to bother when the command is not configured in the Jamal environment. It can come handy in a few situations. For example, you want to use Graphviz to create some nice looking diagrams. Some macros extract the Graphviz dot file from the document and then use Graphviz to create the image. The Jamal processing of the document runs as part of the unit test to ensure that the documentation just as well as the tests are correct and up-to-date.

In this setup you may face the issue that Graphviz is not installed on the continous intergration server. The lack of the application will break the build, since Jamal cannot run the external process. As a workaround you can add the output of Graphviz to the source control and use the option optional. When you build your code on your local system Graphiz is available, configured in your ~/.jamal/settings.properties and works. Whenever you change the graph description in your documentation file, the SVG or PNG of the graph will follow during the build. When the code is comitted to the CVS server the integration server kicks-in, runs the build. The build will see that the Graphviz application is not configured and will ignore the external process.

The example tries to run an external command, which were configured under the symbolic name abrakadabra. It is not configured.

{@io:exec command=abrakadabra optional}

The next example is the extension of the previous one. This time we want to run the non-existent abrakadabra asynchronously, hence the asynch=PRG001 option. Technically the name identifies a user defined macro. However, it results an error if you want to use it as a normal user defined macro. The test checks that the error message belongs to this case and not to the use of an undefined macro.

// using PRG001 as a macro will throw an exception, but not undefined macro
{@io:exec command=abrakadabra optional async=PRG001}{PRG001}

x. io:waitFor

This macro can be used to wait for the completion of a proces started earlier asynchronously. A document may start some external process at an earlier point and needs the result only later. While the external processess runs the document processing can go on and wait for the result when it is needed. The output of the external process cannot be collected from the result of the exec macro. Output of asynchronously started external processes do not appear as the result of the macro. In this case the output is typically redirected to a file and the result can be collected from the file after the waitFor was processed.

The macro waitFor uses a subset of the options of the exec macro. Note that all these options are technically aliases. It means that you cannot use a user defined macro to specify their values. They all have to be specified in the first line of the macro.

  • osOnly, os defines a pattern for the operating system’s name. The execution will only start if the operating system’s name matches the pattern. The pattern is a regular expression. The pattern is matched against the operating system’s name using the Java pattern matching find() method. It means that it is enough to provide a pattern that matches part of the OS name. For example, windows will match Windows 10 and Windows 7 but not Linux. If the pattern is not provided, the execution will start on all operating systems.

  • async, asynch, asynchronous, id, name This option should refer to the name, which was specified in the macro io:exec. The macro will wait for the process that was started with this name to finish. Note that this option has two extra aliases, that do not exist in the macro exec. These are id and name.

  • wait, waitMax, timeOut This option can be used to specify the maximum amount of time in milliseconds to wait for the process to finish. If the process does not finish in the specified time, a BadSyntax exception will be thrown. If this option is not present, the macro will wait for the process to finish without time limit.

  • destroy, kill This option can be used to destroy the process if it has not finished within the specified time. This option can only be used together with the wait option.

  • force, forced This option instructs the macro to destroy the process forcibly. This option can only be used together with the destroy option.

  • optional Use this option if the process was started with the optional option. Using this option will not try to wait for a process, which was not started at the first place.

Examples

The following example starts a one-second sleep as a separate process asynchronous. After that in the next macro it waits for the process to finish.

exec -> sleep
{@io:exec command=EXEC argument=1 asynch=PRG001}{@io:waitFor id=PRG001}

The next example starts a ten-second sleep asynchronously. After that in the next macro it waits for the process to finish with a one seond timeout value (1000ms). It eventually will not finish during this time and then the macro will terminate the external process.

exec -> sleep
{@io:exec command=EXEC argument=10 asynch=PRG001}{@io:waitFor id=PRG001 timeOut=1000 destroy}