diff --git a/.gitignore b/.gitignore index 234f02897..0d7f03b37 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ .RData .Ruserdata docs +inst/doc diff --git a/DESCRIPTION b/DESCRIPTION index 94bd11d3e..3eb773b98 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,7 +1,7 @@ Package: mirai Type: Package Title: Minimalist Async Evaluation Framework for R -Version: 0.10.0.9018 +Version: 0.10.0.9019 Description: Lightweight parallel code execution and distributed computing. Designed for simplicity, a 'mirai' evaluates an R expression asynchronously, on local or network resources, resolving automatically upon completion. @@ -25,4 +25,8 @@ Depends: Imports: nanonext (>= 0.10.1), parallel +Suggests: + knitr, + rmarkdown +VignetteBuilder: knitr RoxygenNote: 7.2.3 diff --git a/NEWS.md b/NEWS.md index 746a463e8..8583b45da 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -# mirai 0.10.0.9018 (development) +# mirai 0.10.0.9019 (development) * Implements an alternative communications backend for R, adding methods for the 'parallel' base package. + Fulfils a request by R Core at R Project Sprint 2023, and requires R >= 4.4 (currently R-devel). @@ -16,6 +16,8 @@ * Reverts the trailing line break added to the end of a 'miraiError' character string. * Deprecates the Deferred Evaluation Pipe `%>>%` in favour of a recommendation to use package `mirai.promises` for performing side effects upon 'mirai' resolution. * Deprecated use of alias `server()` for `daemon()` is retired. +* Document dot parameters for `daemons()` and `dispatcher()` (thanks @krlmlr #79). +* Adds a 'reference' vignette, moving most of the information from the readme. * Requires nanonext >= 0.10.1. # mirai 0.10.0 diff --git a/R/daemons.R b/R/daemons.R index a247ce0f5..8ad761c4f 100644 --- a/R/daemons.R +++ b/R/daemons.R @@ -27,6 +27,8 @@ #' else the low-level approach of distributing tasks to daemons equally. #' #' @inheritParams dispatcher +#' @inheritDotParams dispatcher token:lock +#' @inheritDotParams daemon maxtasks:cleanup #' @param n integer number of daemons to set. #' @param url [default NULL] if specified, the character URL or vector of URLs #' on the host for remote daemons to dial into, including a port accepting diff --git a/R/dispatcher.R b/R/dispatcher.R index 1637b3938..565582aa2 100644 --- a/R/dispatcher.R +++ b/R/dispatcher.R @@ -22,6 +22,7 @@ #' processing, using a FIFO scheduling rule, queuing tasks as required. #' #' @inheritParams daemon +#' @inheritDotParams daemon maxtasks idletime walltime timerstart cleanup #' @param host the character host URL to dial (where tasks are sent from), #' including the port to connect to (and optionally for websockets, a path), #' e.g. 'tcp://192.168.0.2:5555' or 'ws://192.168.0.2:5555/path'. @@ -50,8 +51,6 @@ #' @param pass [default NULL] (required only if the private key supplied to 'tls' #' is encrypted with a password) For security, should be provided through a #' function that returns this value, rather than directly. -#' @param ... additional arguments passed through to \code{\link{daemon}} if -#' launching local daemons i.e. 'url' is not specified. #' @param monitor (for package internal use only) do not set this parameter. #' #' @return Invisible NULL. diff --git a/README.Rmd b/README.Rmd index ab6238e2f..bc2badc95 100644 --- a/README.Rmd +++ b/README.Rmd @@ -36,21 +36,6 @@ Features efficient task scheduling, fast inter-process communications, and Trans [`mirai`](https://doi.org/10.5281/zenodo.7912722) has a tiny pure R code base, relying solely on [`nanonext`](https://doi.org/10.5281/zenodo.7903429), a high-performance binding for the 'NNG' (Nanomsg Next Gen) C library with zero package dependencies.

-### Table of Contents - -1. [Installation](#installation) -2. [Example 1: Compute-intensive Operations](#example-1-compute-intensive-operations) -3. [Example 2: I/O-bound Operations](#example-2-io-bound-operations) -4. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) -5. [Daemons: Local Persistent Processes](#daemons-local-persistent-processes) -6. [Distributed Computing: Remote Daemons](#distributed-computing-remote-daemons) -7. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) -8. [Compute Profiles](#compute-profiles) -9. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) -10. [Integrations with Crew, Targets, Shiny](#integrations-with-crew-targets-shiny) -11. [Thanks](#thanks) -12. [Links](#links) - ### Installation Install the latest release from CRAN: @@ -65,13 +50,7 @@ or the development version from rOpenSci R-universe: install.packages("mirai", repos = "https://shikokuchuo.r-universe.dev") ``` -[« Back to ToC](#table-of-contents) - -### Example 1: Compute-intensive Operations - -Use case: minimise execution times by performing long-running tasks concurrently in separate processes. - -Multiple long computes (model fits etc.) can be performed in parallel on available computing cores. +### Quick Start Use `mirai()` to evaluate an expression asynchronously in a separate, clean R process. @@ -103,6 +82,12 @@ m$data call_mirai(m) ``` +To check whether a mirai has resolved: + +```{r unres} +unresolved(m) +``` + Upon completion, the 'mirai' resolves automatically to the evaluated result. ```{r resolv} @@ -115,421 +100,32 @@ Alternatively, explicitly call and wait for the result using `call_mirai()`. call_mirai(m)$data |> str() ``` -For easy programmatic use of `mirai()`, '.expr' accepts a pre-constructed language object, and also a list of named arguments passed via '.args'. So, the following would be equivalent to the above: - -```{r equiv} -expr <- quote({ - res <- rnorm(n) + m - res / rev(res) -}) - -args <- list(m = runif(1), n = 1e8) - -m <- mirai(.expr = expr, .args = args) - -call_mirai(m)$data |> str() -``` - -[« Back to ToC](#table-of-contents) - -### Example 2: I/O-bound Operations - -Use case: ensure execution flow of the main process is not blocked. - -High-frequency real-time data cannot be written to file/database synchronously without disrupting the execution flow. - -Cache data in memory and use `mirai()` to perform periodic write operations concurrently in a separate process. - -Below, '.args' is used to pass a list of objects already present in the calling environment to the mirai by name. This is an alternative use of '.args', and may be combined with `...` to also pass in `name = value` pairs. - -```{r exec2} -library(mirai) - -x <- rnorm(1e6) -file <- tempfile() - -m <- mirai(write.csv(x, file = file), .args = list(x, file)) -``` - -A 'mirai' object is returned immediately. - -`unresolved()` may be used in control flow statements to perform actions which depend on resolution of the 'mirai', both before and after. - -This means there is no need to actually wait (block) for a 'mirai' to resolve, as the example below demonstrates. - -```{r call2} -# unresolved() queries for resolution itself so no need to use it again within the while loop - -while (unresolved(m)) { - cat("while unresolved\n") - Sys.sleep(0.5) -} - -cat("Write complete:", is.null(m$data)) - -``` - -Now actions which depend on the resolution may be processed, for example the next write. - -[« Back to ToC](#table-of-contents) - -### Example 3: Resilient Pipelines - -Use case: isolating code that can potentially fail in a separate process to ensure continued uptime. - -As part of a data science / machine learning pipeline, iterations of model training may periodically fail for stochastic and uncontrollable reasons (e.g. buggy memory management on graphics cards). - -Running each iteration in a 'mirai' isolates this potentially-problematic code such that even if it does fail, it does not bring down the entire pipeline. - -```{r exec3r} -library(mirai) - -run_iteration <- function(i) { - - if (runif(1) < 0.1) stop("random error\n", call. = FALSE) # simulates a stochastic error rate - sprintf("iteration %d successful\n", i) - -} - -for (i in 1:10) { - - m <- mirai(run_iteration(i), .args = list(run_iteration, i)) - while (is_error_value(call_mirai(m)$data)) { - cat(m$data) - m <- mirai(run_iteration(i), .args = list(run_iteration, i)) - } - cat(m$data) - -} - -``` - -Further, by testing the return value of each 'mirai' for errors, error-handling code is then able to automate recovery and re-attempts, as in the above example. Further details on [error handling](#errors-interrupts-and-timeouts) can be found in the section below. - -The end result is a resilient and fault-tolerant pipeline that minimises downtime by eliminating interruptions of long computes. - -[« Back to ToC](#table-of-contents) - -### Daemons: Local Persistent Processes - -Daemons, or persistent background processes, may be set to receive 'mirai' requests. - -This is potentially more efficient as new processes no longer need to be created on an *ad hoc* basis. - -#### With Dispatcher (default) - -Call `daemons()` specifying the number of daemons to launch. - -```{r daemons} -daemons(6) -``` - -```{r daemons2, include=FALSE} -Sys.sleep(1) -``` - -To view the current status, `status()` provides the number of active connections along with a matrix of statistics for each daemon. - -```{r daemons3} -status() -``` - -The default `dispatcher = TRUE` creates a `dispatcher()` background process that connects to individual daemon processes on the local machine. This ensures that tasks are dispatched efficiently on a first-in first-out (FIFO) basis to daemons for processing. Tasks are queued at the dispatcher and sent to a daemon as soon as it can accept the task for immediate execution. - -Dispatcher uses synchronisation primitives from [`nanonext`](https://doi.org/10.5281/zenodo.7903429), waiting upon rather than polling for tasks, which is efficient both in terms of consuming no resources while waiting, and also being fully synchronised with events (having no latency). - -```{r daemons4} -daemons(0) -``` - -Set the number of daemons to zero to reset. This reverts to the default of creating a new background process for each 'mirai' request. - -#### Without Dispatcher - -Alternatively, specifying `dispatcher = FALSE`, the background daemons connect directly to the host process. - -```{r daemonsq} -daemons(6, dispatcher = FALSE) -``` -```{r daemonsq2, include=FALSE} -Sys.sleep(0.5) -``` - -Requesting the status now shows 6 connections, along with the host URL at `$daemons`. - -```{r daemonsqv} -status() -``` - -This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons remain idle. -The advantage of this approach is that it is low-level and does not require an additional dispatcher process. It is well-suited to working with similar-length tasks, or where the number of concurrent tasks typically does not exceed available daemons. - -```{r daemons5} -daemons(0) -``` - -Set the number of daemons to zero to reset. - -[« Back to ToC](#table-of-contents) +Refer to the '**reference**' vignette for the full package functionality, which includes launching daemons, which are persistent background processes on local or remote machines. -### Distributed Computing: Remote Daemons +### Use with Parallel -The daemons interface may also be used to send tasks for computation to remote daemon processes on the network. +`mirai` provides an alternative communications backend for R's base 'parallel' package. The `make_cluster()` function creates a 'miraiCluster', which leverages the full capabilities of the package, but remains fully compatible with all functions from the parallel pacakge such as `clusterApply` or `parLapply`. -Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections. +Created clusters may also be used in all cases which take a 'cluster' object. For example, [`doParallel`](https://cran.r-project.org/package=doParallel) may be used to register a 'miraiCluster' for use with the package [`foreach`](https://cran.r-project.org/package=foreach). -The examples below use an illustrative local network IP address of '10.75.37.40'. +This functionality, fulfilling a request from R-Core at R Project Sprint 2023, requires R >= 4.4 (currently R-devel). -A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below. - -IPv6 addresses are also supported and must be enclosed in square brackets `[]` to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would be specified as `tcp://[::ffff:a6f:50d]:5555`. - -#### Connecting to Remote Daemons Through Dispatcher - -The default `dispatcher = TRUE` creates a background `dispatcher()` process on the local machine, which listens to a vector of URLs that remote `daemon()` processes dial in to, with each daemon having its own unique URL. - -It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port. - -```{r localqueue} -daemons(n = 4, url = "ws://10.75.37.40:5555") -``` - -Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs. - -Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.: - -```{r vectorqueue, eval=FALSE} -daemons(url = c("ws://10.75.37.40:5566/cpu", "ws://10.75.37.40:5566/gpu", "ws://10.75.37.40:7788/1")) -``` - -Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied. - --- - -On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at: - -``` -Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/1")' -Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/2")' -Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/3")' -Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/4")' - -``` +### Use with Crew and Targets -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. +The [`crew`](https://wlandau.github.io/crew/) package is a distributed worker-launcher that provides an R6-based interface extending `mirai` to different distributed computing platforms, from traditional clusters to cloud services. -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.37.100 accepts SSH connections over port 22: - -```{r launchremote, eval=FALSE} -launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.37.100", .)) -``` -```{r launchremotereal, echo=FALSE} -launch_remote(1:4) -``` - -The returned vector comprises the shell commands executed on the remote machine. - --- - -Requesting status, on the host machine: - -```{r remotev2} -status() -``` - -As per the local case, `$connections` shows the single connection to dispatcher, however `$daemons` now provides a matrix of statistics for the remote daemons. - -- `i` index number. -- `online` shows as 1 when there is an active connection, or else 0 if a daemon has yet to connect or has disconnected. -- `instance` increments by 1 every time there is a new connection at a URL. This counter is designed to track new daemon instances connecting after previous ones have ended (due to time-outs etc.). The count becomes negative immediately after a URL is regenerated by `saisei()`, but increments again once a new daemon connects. -- `assigned` shows the cumulative number of tasks assigned to the daemon. -- `complete` shows the cumulative number of tasks completed by the daemon. - -Dispatcher automatically adjusts to the number of daemons actually connected. Hence it is possible to dynamically scale up or down the number of daemons according to requirements (limited to the 'n' URLs assigned). - -To reset all connections and revert to default behaviour: - -```{r reset2} -daemons(0) -``` - -Closing the connection causes the dispatcher to exit automatically, and in turn all connected daemons when their respective connections with the dispatcher are terminated. - -#### Connecting to Remote Daemons Directly - -By specifying `dispatcher = FALSE`, remote daemons connect directly to the host process. The host listens at a single URL, and distributes tasks to all connected daemons. - -```{r remote, eval=FALSE} -daemons(url = "tcp://10.75.37.40:0", dispatcher = FALSE) -``` - -Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example: - -```{r remotealt} -daemons(url = "tcp://:0", dispatcher = FALSE) -``` - -Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. - --- - -On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks: - -``` -Rscript -e 'mirai::daemon("tcp://10.75.37.40:0")' -``` -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. - -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.37.100 accepts SSH connections over port 22: - -```{r launchremoteu, eval=FALSE} -launch_remote("tcp://10.75.37.40:0", command = "ssh", args = c("-p 22 10.75.37.100", .)) -``` -```{r launchremoteureal, echo=FALSE} -launch_remote("tcp://10.75.37.40:0") -``` - -The returned vector comprises the shell commands executed on the remote machine. - --- - -The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. - -`$connections` will show the actual number of connected daemons. - -```{r remotev} -status() -``` - -To reset all connections and revert to default behaviour: - -```{r reset} -daemons(0) -``` - -This causes all connected daemons to exit automatically. - -[« Back to ToC](#table-of-contents) - -### Distributed Computing: TLS Secure Connections - -TLS is available as an option to secure communications from the local machine to remote daemons. - -#### Zero-configuration - -An automatic zero-configuration default is implemented. Simply specify a secure URL of the form `wss://` or `tls+tcp://` when setting daemons. For example, on the IPv6 loopback address: - -```{r tlsremote} -daemons(n = 4, url = "wss://[::1]:5555") -``` - -Single-use keys and certificates are automatically generated and configured, without requiring any further intervention. The private key is always retained on the host machine and never transmitted. - -The generated self-signed certificate is available via `launch_remote()`. This function conveniently constructs the full shell command to launch a daemon, including the correctly specified 'tls' argument to `daemon()`. - -```{r launch_remote} -launch_remote(1) -``` - -The return value may be deployed manually on a remote machine by unescaping the double quotes around the call to `"mirai::daemon()"`, or directly via SSH or a resource manager by additionally specifying 'command' and 'args' to `launch_remote()`. - -```{r tlsclose, include=FALSE} -daemons(0) -``` - -#### CA Signed Certificates - -As an alternative to the zero-configuration option, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to your organisation. - -1. Generate a private key and CSR. The following resources describe how to do so: - -- using Mbed TLS: -- using OpenSSL: (Chapter 1.2 Key and Certificate Management) - -2. Send or provide the generated CSR to the CA for it to sign a new TLS certificate. - -- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, your TLS library should usually provide conversion utilities. -- Check also that your private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. - -3. When setting daemons, the TLS certificate and private key should be provided to the 'tls' argument of `daemons()`. - -- If the certificate and private key have been imported as character strings `cert` and `key` respectively, then the 'tls' argument may be specified as the character vector `c(cert, key)`. -- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this new file may be provided to the 'tls' argument. - -4. When launching daemons, the certificate chain to the CA should be supplied to the 'tls' argument of `daemon()` or `launch_remote()`. - -- The certificate chain should comprise multiple certificates, each between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers. The first one should be the newly-generated TLS certificate, the same supplied to `daemons()`, and the final one should be a CA root certificate. -- These are the only certificates required if your certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with your TLS certificate and ends with the certificate of the CA. -- If these are concatenated together as a single character string `certchain` (and assuming no certificate revocation list), then the character vector `c(certchain, "")` may be supplied to the relevant 'tls' argument. -- Alternatively, if these are written to a file (and the file replicated on the remote machines), then the 'tls' argument may also be specified as a path/filename (assuming these are the same on each machine). - -[« Back to ToC](#table-of-contents) - -### Compute Profiles - -The `daemons()` interface also allows the specification of compute profiles for managing tasks with heterogeneous compute requirements: - -- send tasks to different daemons or clusters of daemons with the appropriate specifications (in terms of CPUs / memory / GPU / accelerators etc.) -- split tasks between local and remote computation - -Simply specify the argument `.compute` when calling `daemons()` with a profile name (which is 'default' for the default profile). The daemons settings are saved under the named profile. - -To create a 'mirai' task using a specific compute profile, specify the '.compute' argument to `mirai()`, which defaults to the 'default' compute profile. - -Similarly, functions such as `status()`, `launch_local()` or `launch_remote()` should be specified with the desired '.compute' argument. - -[« Back to ToC](#table-of-contents) - -### Errors, Interrupts and Timeouts - -If execution in a mirai fails, the error message is returned as a character string of class 'miraiError' and 'errorValue' to facilitate debugging. `is_mirai_error()` may be used to test for mirai execution errors. - -```{r errorexample} -m1 <- mirai(stop("occurred with a custom message", call. = FALSE)) -call_mirai(m1)$data - -m2 <- mirai(mirai::mirai()) -call_mirai(m2)$data - -is_mirai_error(m2$data) -is_error_value(m2$data) -``` - -If a daemon instance is sent a user interrupt, the mirai will resolve to an empty character string of class 'miraiInterrupt' and 'errorValue'. `is_mirai_interrupt()` may be used to test for such interrupts. - -```{r interruptexample} -is_mirai_interrupt(m2$data) -``` - -If execution of a mirai surpasses the timeout set via the '.timeout' argument, the mirai will resolve to an 'errorValue'. This can, amongst other things, guard against mirai processes that have the potential to hang and never return. - -```{r timeouts} -m3 <- mirai(nanonext::msleep(1000), .timeout = 500) -call_mirai(m3)$data - -is_mirai_error(m3$data) -is_mirai_interrupt(m3$data) -is_error_value(m3$data) -``` - -`is_error_value()` tests for all mirai execution errors, user interrupts and timeouts. - -[« Back to ToC](#table-of-contents) - -### Integrations with Crew, Targets, Shiny - -The [`crew`](https://wlandau.github.io/crew/) package is a distributed worker-launcher that provides an R6-based interface extending `mirai` to different distributed computing platforms, from traditional clusters to cloud services. The [`crew.cluster`](https://wlandau.github.io/crew.cluster/) package is a plug-in that enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and SLURM. +The [`crew.cluster`](https://wlandau.github.io/crew.cluster/) package is a plug-in that enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and SLURM. [`targets`](https://docs.ropensci.org/targets/), a Make-like pipeline tool for statistics and data science, has integrated and adopted [`crew`](https://wlandau.github.io/crew/) as its predominant high-performance computing backend. -`mirai` can also serve as the backend for enterprise asynchronous [`shiny`](https://cran.r-project.org/package=shiny) applications in one of two ways: +### Use with Shiny -1. [`mirai.promises`](https://shikokuchuo.net/mirai.promises/), which enables a 'mirai' to be used interchangeably with a 'promise' in [`shiny`](https://cran.r-project.org/package=shiny) or [`plumber`](https://cran.r-project.org/package=plumber) pipelines; or +`mirai` can also serve as the backend for enterprise asynchronous [`shiny`](https://cran.r-project.org/package=shiny) applications through either: -2. [`crew`](https://wlandau.github.io/crew/) provides an interface that makes it easy to deploy `mirai` for [`shiny`](https://cran.r-project.org/package=shiny). The package provides a [Shiny vignette](https://wlandau.github.io/crew/articles/shiny.html) with tutorial and sample code for this purpose. +[`mirai.promises`](https://shikokuchuo.net/mirai.promises/), which enables a 'mirai' to be used interchangeably with a 'promise', allowing side-effects to be performed upon resolution of a 'mirai'; or -[« Back to ToC](#table-of-contents) +[`crew`](https://wlandau.github.io/crew/) provides an interface that makes it easy to deploy `mirai` for [`shiny`](https://cran.r-project.org/package=shiny). The package provides a [Shiny vignette](https://wlandau.github.io/crew/articles/shiny.html) with tutorial and sample code for this purpose. ### Thanks diff --git a/README.md b/README.md index dd0655949..87db5bc91 100644 --- a/README.md +++ b/README.md @@ -29,26 +29,6 @@ base, relying solely on binding for the ‘NNG’ (Nanomsg Next Gen) C library with zero package dependencies.

-### Table of Contents - -1. [Installation](#installation) -2. [Example 1: Compute-intensive - Operations](#example-1-compute-intensive-operations) -3. [Example 2: I/O-bound Operations](#example-2-io-bound-operations) -4. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) -5. [Daemons: Local Persistent - Processes](#daemons-local-persistent-processes) -6. [Distributed Computing: Remote - Daemons](#distributed-computing-remote-daemons) -7. [Distributed Computing: TLS Secure - Connections](#distributed-computing-tls-secure-connections) -8. [Compute Profiles](#compute-profiles) -9. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) -10. [Integrations with Crew, Targets, - Shiny](#integrations-with-crew-targets-shiny) -11. [Thanks](#thanks) -12. [Links](#links) - ### Installation Install the latest release from CRAN: @@ -63,15 +43,7 @@ or the development version from rOpenSci R-universe: install.packages("mirai", repos = "https://shikokuchuo.r-universe.dev") ``` -[« Back to ToC](#table-of-contents) - -### Example 1: Compute-intensive Operations - -Use case: minimise execution times by performing long-running tasks -concurrently in separate processes. - -Multiple long computes (model fits etc.) can be performed in parallel on -available computing cores. +### Quick Start Use `mirai()` to evaluate an expression asynchronously in a separate, clean R process. @@ -106,12 +78,19 @@ m$data #> 'unresolved' logi NA ``` +To check whether a mirai has resolved: + +``` r +unresolved(m) +#> [1] FALSE +``` + Upon completion, the ‘mirai’ resolves automatically to the evaluated result. ``` r m$data |> str() -#> num [1:100000000] 0.601 2.251 -0.47 0.296 0.271 ... +#> num [1:100000000] 1.362 -1.846 0.473 0.666 -3.015 ... ``` Alternatively, explicitly call and wait for the result using @@ -119,627 +98,39 @@ Alternatively, explicitly call and wait for the result using ``` r call_mirai(m)$data |> str() -#> num [1:100000000] 0.601 2.251 -0.47 0.296 0.271 ... -``` - -For easy programmatic use of `mirai()`, ‘.expr’ accepts a -pre-constructed language object, and also a list of named arguments -passed via ‘.args’. So, the following would be equivalent to the above: - -``` r -expr <- quote({ - res <- rnorm(n) + m - res / rev(res) -}) - -args <- list(m = runif(1), n = 1e8) - -m <- mirai(.expr = expr, .args = args) - -call_mirai(m)$data |> str() -#> num [1:100000000] 6.42 3.24 0.64 2.76 1.39 ... -``` - -[« Back to ToC](#table-of-contents) - -### Example 2: I/O-bound Operations - -Use case: ensure execution flow of the main process is not blocked. - -High-frequency real-time data cannot be written to file/database -synchronously without disrupting the execution flow. - -Cache data in memory and use `mirai()` to perform periodic write -operations concurrently in a separate process. - -Below, ‘.args’ is used to pass a list of objects already present in the -calling environment to the mirai by name. This is an alternative use of -‘.args’, and may be combined with `...` to also pass in `name = value` -pairs. - -``` r -library(mirai) - -x <- rnorm(1e6) -file <- tempfile() - -m <- mirai(write.csv(x, file = file), .args = list(x, file)) -``` - -A ‘mirai’ object is returned immediately. - -`unresolved()` may be used in control flow statements to perform actions -which depend on resolution of the ‘mirai’, both before and after. - -This means there is no need to actually wait (block) for a ‘mirai’ to -resolve, as the example below demonstrates. - -``` r -# unresolved() queries for resolution itself so no need to use it again within the while loop - -while (unresolved(m)) { - cat("while unresolved\n") - Sys.sleep(0.5) -} -#> while unresolved -#> while unresolved - -cat("Write complete:", is.null(m$data)) -#> Write complete: TRUE -``` - -Now actions which depend on the resolution may be processed, for example -the next write. - -[« Back to ToC](#table-of-contents) - -### Example 3: Resilient Pipelines - -Use case: isolating code that can potentially fail in a separate process -to ensure continued uptime. - -As part of a data science / machine learning pipeline, iterations of -model training may periodically fail for stochastic and uncontrollable -reasons (e.g. buggy memory management on graphics cards). - -Running each iteration in a ‘mirai’ isolates this -potentially-problematic code such that even if it does fail, it does not -bring down the entire pipeline. - -``` r -library(mirai) - -run_iteration <- function(i) { - - if (runif(1) < 0.1) stop("random error\n", call. = FALSE) # simulates a stochastic error rate - sprintf("iteration %d successful\n", i) - -} - -for (i in 1:10) { - - m <- mirai(run_iteration(i), .args = list(run_iteration, i)) - while (is_error_value(call_mirai(m)$data)) { - cat(m$data) - m <- mirai(run_iteration(i), .args = list(run_iteration, i)) - } - cat(m$data) - -} -#> iteration 1 successful -#> iteration 2 successful -#> iteration 3 successful -#> iteration 4 successful -#> iteration 5 successful -#> iteration 6 successful -#> iteration 7 successful -#> iteration 8 successful -#> iteration 9 successful -#> Error: random error -#> iteration 10 successful -``` - -Further, by testing the return value of each ‘mirai’ for errors, -error-handling code is then able to automate recovery and re-attempts, -as in the above example. Further details on [error -handling](#errors-interrupts-and-timeouts) can be found in the section -below. - -The end result is a resilient and fault-tolerant pipeline that minimises -downtime by eliminating interruptions of long computes. - -[« Back to ToC](#table-of-contents) - -### Daemons: Local Persistent Processes - -Daemons, or persistent background processes, may be set to receive -‘mirai’ requests. - -This is potentially more efficient as new processes no longer need to be -created on an *ad hoc* basis. - -#### With Dispatcher (default) - -Call `daemons()` specifying the number of daemons to launch. - -``` r -daemons(6) -#> [1] 6 -``` - -To view the current status, `status()` provides the number of active -connections along with a matrix of statistics for each daemon. - -``` r -status() -#> $connections -#> [1] 1 -#> -#> $daemons -#> i online instance assigned complete -#> abstract://988b6c5548b89873daae7d6b 1 1 1 0 0 -#> abstract://f968e887dd6aafb09af3f9ec 2 1 1 0 0 -#> abstract://285a8ea0c175ea5b676ebca8 3 1 1 0 0 -#> abstract://f1b2bcd7f93e7fb829970f23 4 1 1 0 0 -#> abstract://6e16a65c5b1764e6a4431e4b 5 1 1 0 0 -#> abstract://3843671f338e8c28f8c469ad 6 1 1 0 0 -``` - -The default `dispatcher = TRUE` creates a `dispatcher()` background -process that connects to individual daemon processes on the local -machine. This ensures that tasks are dispatched efficiently on a -first-in first-out (FIFO) basis to daemons for processing. Tasks are -queued at the dispatcher and sent to a daemon as soon as it can accept -the task for immediate execution. - -Dispatcher uses synchronisation primitives from -[`nanonext`](https://doi.org/10.5281/zenodo.7903429), waiting upon -rather than polling for tasks, which is efficient both in terms of -consuming no resources while waiting, and also being fully synchronised -with events (having no latency). - -``` r -daemons(0) -#> [1] 0 -``` - -Set the number of daemons to zero to reset. This reverts to the default -of creating a new background process for each ‘mirai’ request. - -#### Without Dispatcher - -Alternatively, specifying `dispatcher = FALSE`, the background daemons -connect directly to the host process. - -``` r -daemons(6, dispatcher = FALSE) -#> [1] 6 -``` - -Requesting the status now shows 6 connections, along with the host URL -at `$daemons`. - -``` r -status() -#> $connections -#> [1] 6 -#> -#> $daemons -#> [1] "abstract://3a21cdc05821276862216ae1" -``` - -This implementation sends tasks immediately, and ensures that tasks are -evenly-distributed amongst daemons. This means that optimal scheduling -is not guaranteed as the duration of tasks cannot be known *a priori*. -As an example, tasks could be queued at a daemon behind a long-running -task, whilst other daemons remain idle. - -The advantage of this approach is that it is low-level and does not -require an additional dispatcher process. It is well-suited to working -with similar-length tasks, or where the number of concurrent tasks -typically does not exceed available daemons. - -``` r -daemons(0) -#> [1] 0 -``` - -Set the number of daemons to zero to reset. - -[« Back to ToC](#table-of-contents) - -### Distributed Computing: Remote Daemons - -The daemons interface may also be used to send tasks for computation to -remote daemon processes on the network. - -Call `daemons()` specifying ‘url’ as a character string the host network -address and a port that is able to accept incoming connections. - -The examples below use an illustrative local network IP address of -‘10.75.37.40’. - -A port on the host machine also needs to be open and available for -inbound connections from the local network, illustratively ‘5555’ in the -examples below. - -IPv6 addresses are also supported and must be enclosed in square -brackets `[]` to avoid confusion with the final colon separating the -port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would -be specified as `tcp://[::ffff:a6f:50d]:5555`. - -#### Connecting to Remote Daemons Through Dispatcher - -The default `dispatcher = TRUE` creates a background `dispatcher()` -process on the local machine, which listens to a vector of URLs that -remote `daemon()` processes dial in to, with each daemon having its own -unique URL. - -It is recommended to use a websocket URL starting `ws://` instead of TCP -in this scenario (used interchangeably with `tcp://`). A websocket URL -supports a path after the port number, which can be made unique for each -daemon. In this way a dispatcher can connect to an arbitrary number of -daemons over a single port. - -``` r -daemons(n = 4, url = "ws://10.75.37.40:5555") -#> [1] 4 +#> num [1:100000000] 1.362 -1.846 0.473 0.666 -3.015 ... ``` -Above, a single URL was supplied, along with `n = 4` to specify that the -dispatcher should listen at 4 URLs. In such a case, an integer sequence -is automatically appended to the path `/1` through `/4` to produce these -URLs. +Refer to the ‘**reference**’ vignette for the full package +functionality, which includes launching daemons, which are persistent +background processes on local or remote machines. -Alternatively, supplying a vector of URLs allows the use of arbitrary -port numbers / paths, e.g.: +### Use with Parallel -``` r -daemons(url = c("ws://10.75.37.40:5566/cpu", "ws://10.75.37.40:5566/gpu", "ws://10.75.37.40:7788/1")) -``` - -Above, ‘n’ is not specified, in which case its value is inferred from -the length of the ‘url’ vector supplied. - -– - -On the remote resource, `daemon()` may be called from an R session, or -directly from a shell using Rscript. Each daemon instance should dial -into one of the unique URLs that the dispatcher is listening at: - - Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/1")' - Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/2")' - Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/3")' - Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/4")' - -Note that `daemons()` should be set up on the host machine before -launching `daemon()` on remote resources, otherwise the daemon instances -will exit if a connection is not immediately available. Alternatively, -specifying `daemon(asyncdial = TRUE)` will allow daemons to wait -(indefinitely) for a connection to become available. - -`launch_remote()` may also be used to launch daemons directly on a -remote machine. For example, if the remote machine at 10.75.37.100 -accepts SSH connections over port 22: - -``` r -launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.37.100", .)) -#> [1] "Rscript -e \"mirai::daemon('ws://10.75.37.40:5555/1',rs=c(10407,234847007,-1443550508,-1219227707,585277890,326394459,-544448032))\"" -#> [2] "Rscript -e \"mirai::daemon('ws://10.75.37.40:5555/2',rs=c(10407,855496323,1126561919,560666770,141328549,1513462613,-349875403))\"" -#> [3] "Rscript -e \"mirai::daemon('ws://10.75.37.40:5555/3',rs=c(10407,1901043322,1483328582,81985270,1276055119,-1503907136,-404210225))\"" -#> [4] "Rscript -e \"mirai::daemon('ws://10.75.37.40:5555/4',rs=c(10407,668343214,-722105549,-1445000249,515588687,1646507310,1828364408))\"" -``` - -The returned vector comprises the shell commands executed on the remote -machine. - -– - -Requesting status, on the host machine: - -``` r -status() -#> $connections -#> [1] 1 -#> -#> $daemons -#> i online instance assigned complete -#> ws://10.75.37.40:5555/1 1 1 1 0 0 -#> ws://10.75.37.40:5555/2 2 1 1 0 0 -#> ws://10.75.37.40:5555/3 3 1 1 0 0 -#> ws://10.75.37.40:5555/4 4 1 1 0 0 -``` - -As per the local case, `$connections` shows the single connection to -dispatcher, however `$daemons` now provides a matrix of statistics for -the remote daemons. - -- `i` index number. -- `online` shows as 1 when there is an active connection, or else 0 if a - daemon has yet to connect or has disconnected. -- `instance` increments by 1 every time there is a new connection at a - URL. This counter is designed to track new daemon instances connecting - after previous ones have ended (due to time-outs etc.). The count - becomes negative immediately after a URL is regenerated by `saisei()`, - but increments again once a new daemon connects. -- `assigned` shows the cumulative number of tasks assigned to the - daemon. -- `complete` shows the cumulative number of tasks completed by the - daemon. - -Dispatcher automatically adjusts to the number of daemons actually -connected. Hence it is possible to dynamically scale up or down the -number of daemons according to requirements (limited to the ‘n’ URLs -assigned). - -To reset all connections and revert to default behaviour: - -``` r -daemons(0) -#> [1] 0 -``` - -Closing the connection causes the dispatcher to exit automatically, and -in turn all connected daemons when their respective connections with the -dispatcher are terminated. - -#### Connecting to Remote Daemons Directly - -By specifying `dispatcher = FALSE`, remote daemons connect directly to -the host process. The host listens at a single URL, and distributes -tasks to all connected daemons. +`mirai` provides an alternative communications backend for R’s base +‘parallel’ package. The `make_cluster()` function creates a +‘miraiCluster’, which leverages the full capabilities of the package, +but remains fully compatible with all functions from the parallel +pacakge such as `clusterApply` or `parLapply`. -``` r -daemons(url = "tcp://10.75.37.40:0", dispatcher = FALSE) -``` - -Alternatively, simply supply a colon followed by the port number to -listen on all interfaces on the local host, for example: - -``` r -daemons(url = "tcp://:0", dispatcher = FALSE) -#> [1] "tcp://:35989" -``` - -Note that above, the port number is specified as zero. This is a -wildcard value that will automatically cause a free ephemeral port to be -assigned. The actual assigned port is provided in the return value of -the call, or it may be queried at any time via `status()`. - -– - -On the network resource, `daemon()` may be called from an R session, or -an Rscript invocation from a shell. This sets up a remote daemon process -that connects to the host URL and receives tasks: - - Rscript -e 'mirai::daemon("tcp://10.75.37.40:35989")' - -Note that `daemons()` should be set up on the host machine before -launching `daemon()` on remote resources, otherwise the daemon instances -will exit if a connection is not immediately available. Alternatively, -specifying `daemon(asyncdial = TRUE)` will allow daemons to wait -(indefinitely) for a connection to become available. - -`launch_remote()` may also be used to launch daemons directly on a -remote machine. For example, if the remote machine at 10.75.37.100 -accepts SSH connections over port 22: +Created clusters may also be used in all cases which take a ‘cluster’ +object. For example, +[`doParallel`](https://cran.r-project.org/package=doParallel) may be +used to register a ‘miraiCluster’ for use with the package +[`foreach`](https://cran.r-project.org/package=foreach). -``` r -launch_remote("tcp://10.75.37.40:35989", command = "ssh", args = c("-p 22 10.75.37.100", .)) -#> [1] "Rscript -e \"mirai::daemon('tcp://10.75.37.40:35989',rs=c(10407,-1375240495,1010969182,-947866809,-26137892,-1431798227,-1249750262))\"" -``` - -The returned vector comprises the shell commands executed on the remote -machine. - -– - -The number of daemons connecting to the host URL is not limited and -network resources may be added or removed at any time, with tasks -automatically distributed to all connected daemons. - -`$connections` will show the actual number of connected daemons. +This functionality, fulfilling a request from R-Core at R Project Sprint +2023, requires R \>= 4.4 (currently R-devel). -``` r -status() -#> $connections -#> [1] 1 -#> -#> $daemons -#> [1] "tcp://:35989" -``` - -To reset all connections and revert to default behaviour: - -``` r -daemons(0) -#> [1] 0 -``` - -This causes all connected daemons to exit automatically. - -[« Back to ToC](#table-of-contents) - -### Distributed Computing: TLS Secure Connections - -TLS is available as an option to secure communications from the local -machine to remote daemons. - -#### Zero-configuration - -An automatic zero-configuration default is implemented. Simply specify a -secure URL of the form `wss://` or `tls+tcp://` when setting daemons. -For example, on the IPv6 loopback address: - -``` r -daemons(n = 4, url = "wss://[::1]:5555") -#> [1] 4 -``` - -Single-use keys and certificates are automatically generated and -configured, without requiring any further intervention. The private key -is always retained on the host machine and never transmitted. - -The generated self-signed certificate is available via -`launch_remote()`. This function conveniently constructs the full shell -command to launch a daemon, including the correctly specified ‘tls’ -argument to `daemon()`. - -``` r -launch_remote(1) -#> [1] "Rscript -e \"mirai::daemon('wss://[::1]:5555/1',tls=c('-----BEGIN CERTIFICATE-----\nMIIFLTCCAxWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMQwwCgYDVQQDDAM6OjEx\nETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAxMDEwMDAwMDBa\nFw0zMDEyMzEyMzU5NTlaMC4xDDAKBgNVBAMMAzo6MTERMA8GA1UECgwITmFub25l\neHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA\nxx5G9OjsMAUgfKcggxLOUVWdC6sdlCQzDbzOrvEHghwphkt924pYaNgS8UKMnb46\nUFPCPfv1YtJEaUR87hLXBASnAHqvs4akXvyByI2LIREz58/q46wRbuzJq9OnbdiO\nOkcMKX423p5pRNmAbXMsJK3gPgTnr6rd54R4O8a34Dw1ZRdGKXXYYChs/CYrs4bf\niMj9RUBUGYdw24KzAybdaMTMqpysIM5D3K6sGY41n5E7ElSCfrNergpIZ9sG4okK\nekTof8EOrQSJIJ7ni+NeH3rYmcmaD9cgFtyWdPuuHWBfcWcHu6TKlM18GstKw45g\nQTUE79N2/5cnsUc9qq0ce7XaSkwdyvS1pyxaahrIft2fsttWN6v3BlHeoWs/VCRr\nEkPPMJtAJK1dql73l8m0a9siYu7ScKkY0TlKac5AFcQyfL8Tkcj9EtGTj6jxTqOP\ns5RrjehvNABdNQhCq5znmoedVuNBN9oNkQKGuKb5Tsj320oEwtgccXrIZDbmsWyZ\ne0S1EJYo4PWBM63xnlJpcwg5IOd8MFfflfjLCYU+LUnIR1ynKcAChrebNYn4+vBd\njNHjK2Ka0cFOpEl8M4YUV1acK0wjn6A8lHqXhqewihGzNJnZsI8Ltl7MP6oZL6kx\nKWHi99P6PNqCKXQNui8lPzzXuhuCP/TCmIoufN23OaUCAwEAAaNWMFQwEgYDVR0T\nAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU2onzopjVaToXnhSPej8sfKIBRJAwHwYD\nVR0jBBgwFoAU2onzopjVaToXnhSPej8sfKIBRJAwDQYJKoZIhvcNAQELBQADggIB\nADWe5WSIMpj9dvOkb/mPx0DP+XIlHwh2FF9M1CDU20Tal2CikIFmbcXv4H27TCyg\nepB3Y++UNrptl00RwAkd/HywRXSZv053UOFqPs0BEp+kIH7lI1Ouv9adohD+f/QL\nLMFntX7Rgh4UML55hcLK2DyeMRUKlxrjtizB3eo2iZQJ71iFNO0VUbRfqboZcBol\nNIL+InGNoMbhqRecCPfg5RPsQmsp/SVAsuNa3v01A9fBsmz6O0C4C77j62gk4nUt\nPi3YU8zuoHDFuQcjHPRxt2o5svVxxxneGpmqgFg71uuLGesxa/HfOuQc9aO3kaDI\nvZKyAC0Kg/0hkEA5mNI7BGUrMSeTE4virjL+D8iiej7VpR2nntWbJhNWZamAgSZs\nFw9o2lCP17m978hk8YWSWfgG81QBkQoDEANMq5EY+fp6+G6CLs0gIRJSo289xwcT\n2TbPj8/KBJBbSZdWTd8xwtaqwg8YO/dx3OJG1k+hEcGnic9WhvE6Z5LzIm+kZPc3\nC7HjyOJYkoxqjB8SR4l3u1fmn3QX7jlcOhMj0SKXOLGFwztlehk9LPfBhuYQUtqG\n7XcBBcfqtcTS5KDXkHzuC9zTdFwerozW5hygY0KoAfTYHdjM6CmVmtVBvJ6PvuNn\nG6hl03vdqrp9FOis/D4fTxhtRqQbOBmnY1E2lNIWHrl5\n-----END CERTIFICATE-----\n',''),rs=c(10407,-885815674,-593655985,827546948,415376245,-759671374,1873324427))\"" -``` - -The return value may be deployed manually on a remote machine by -unescaping the double quotes around the call to `"mirai::daemon()"`, or -directly via SSH or a resource manager by additionally specifying -‘command’ and ‘args’ to `launch_remote()`. - -#### CA Signed Certificates - -As an alternative to the zero-configuration option, a certificate may -also be generated via a Certificate Signing Request (CSR) to a -Certificate Authority (CA), which may be a public CA or a CA internal to -your organisation. - -1. Generate a private key and CSR. The following resources describe how - to do so: - -- using Mbed TLS: - -- using OpenSSL: - (Chapter - 1.2 Key and Certificate Management) - -2. Send or provide the generated CSR to the CA for it to sign a new TLS - certificate. - -- The received certificate should comprise a block of cipher text - between the markers `-----BEGIN CERTIFICATE-----` and - `-----END CERTIFICATE-----`. Make sure to request the certificate in - the PEM format. If only available in other formats, your TLS library - should usually provide conversion utilities. -- Check also that your private key is a block of cipher text between the - markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. - -3. When setting daemons, the TLS certificate and private key should be - provided to the ‘tls’ argument of `daemons()`. - -- If the certificate and private key have been imported as character - strings `cert` and `key` respectively, then the ‘tls’ argument may be - specified as the character vector `c(cert, key)`. -- Alternatively, the certificate may be copied to a new text file, with - the private key appended, in which case the path/filename of this new - file may be provided to the ‘tls’ argument. - -4. When launching daemons, the certificate chain to the CA should be - supplied to the ‘tls’ argument of `daemon()` or `launch_remote()`. - -- The certificate chain should comprise multiple certificates, each - between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` - markers. The first one should be the newly-generated TLS certificate, - the same supplied to `daemons()`, and the final one should be a CA - root certificate. -- These are the only certificates required if your certificate was - signed directly by a CA. If not, then the intermediate certificates - should be included in a certificate chain that starts with your TLS - certificate and ends with the certificate of the CA. -- If these are concatenated together as a single character string - `certchain` (and assuming no certificate revocation list), then the - character vector `c(certchain, "")` may be supplied to the relevant - ‘tls’ argument. -- Alternatively, if these are written to a file (and the file replicated - on the remote machines), then the ‘tls’ argument may also be specified - as a path/filename (assuming these are the same on each machine). - -[« Back to ToC](#table-of-contents) - -### Compute Profiles - -The `daemons()` interface also allows the specification of compute -profiles for managing tasks with heterogeneous compute requirements: - -- send tasks to different daemons or clusters of daemons with the - appropriate specifications (in terms of CPUs / memory / GPU / - accelerators etc.) -- split tasks between local and remote computation - -Simply specify the argument `.compute` when calling `daemons()` with a -profile name (which is ‘default’ for the default profile). The daemons -settings are saved under the named profile. - -To create a ‘mirai’ task using a specific compute profile, specify the -‘.compute’ argument to `mirai()`, which defaults to the ‘default’ -compute profile. - -Similarly, functions such as `status()`, `launch_local()` or -`launch_remote()` should be specified with the desired ‘.compute’ -argument. - -[« Back to ToC](#table-of-contents) - -### Errors, Interrupts and Timeouts - -If execution in a mirai fails, the error message is returned as a -character string of class ‘miraiError’ and ‘errorValue’ to facilitate -debugging. `is_mirai_error()` may be used to test for mirai execution -errors. - -``` r -m1 <- mirai(stop("occurred with a custom message", call. = FALSE)) -call_mirai(m1)$data -#> 'miraiError' chr Error: occurred with a custom message - -m2 <- mirai(mirai::mirai()) -call_mirai(m2)$data -#> 'miraiError' chr Error in mirai::mirai(): missing expression, perhaps wrap in {}? - -is_mirai_error(m2$data) -#> [1] TRUE -is_error_value(m2$data) -#> [1] TRUE -``` - -If a daemon instance is sent a user interrupt, the mirai will resolve to -an empty character string of class ‘miraiInterrupt’ and ‘errorValue’. -`is_mirai_interrupt()` may be used to test for such interrupts. - -``` r -is_mirai_interrupt(m2$data) -#> [1] FALSE -``` - -If execution of a mirai surpasses the timeout set via the ‘.timeout’ -argument, the mirai will resolve to an ‘errorValue’. This can, amongst -other things, guard against mirai processes that have the potential to -hang and never return. - -``` r -m3 <- mirai(nanonext::msleep(1000), .timeout = 500) -call_mirai(m3)$data -#> 'errorValue' int 5 | Timed out - -is_mirai_error(m3$data) -#> [1] FALSE -is_mirai_interrupt(m3$data) -#> [1] FALSE -is_error_value(m3$data) -#> [1] TRUE -``` - -`is_error_value()` tests for all mirai execution errors, user interrupts -and timeouts. - -[« Back to ToC](#table-of-contents) - -### Integrations with Crew, Targets, Shiny +### Use with Crew and Targets The [`crew`](https://wlandau.github.io/crew/) package is a distributed worker-launcher that provides an R6-based interface extending `mirai` to different distributed computing platforms, from traditional clusters to -cloud services. The -[`crew.cluster`](https://wlandau.github.io/crew.cluster/) package is a -plug-in that enables mirai-based workflows on traditional +cloud services. + +The [`crew.cluster`](https://wlandau.github.io/crew.cluster/) package is +a plug-in that enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and SLURM. @@ -748,24 +139,22 @@ tool for statistics and data science, has integrated and adopted [`crew`](https://wlandau.github.io/crew/) as its predominant high-performance computing backend. -`mirai` can also serve as the backend for enterprise asynchronous -[`shiny`](https://cran.r-project.org/package=shiny) applications in one -of two ways: - -1. [`mirai.promises`](https://shikokuchuo.net/mirai.promises/), which - enables a ‘mirai’ to be used interchangeably with a ‘promise’ in - [`shiny`](https://cran.r-project.org/package=shiny) or - [`plumber`](https://cran.r-project.org/package=plumber) pipelines; - or - -2. [`crew`](https://wlandau.github.io/crew/) provides an interface that - makes it easy to deploy `mirai` for - [`shiny`](https://cran.r-project.org/package=shiny). The package - provides a [Shiny - vignette](https://wlandau.github.io/crew/articles/shiny.html) with - tutorial and sample code for this purpose. +### Use with Shiny -[« Back to ToC](#table-of-contents) +`mirai` can also serve as the backend for enterprise asynchronous +[`shiny`](https://cran.r-project.org/package=shiny) applications through +either: + +[`mirai.promises`](https://shikokuchuo.net/mirai.promises/), which +enables a ‘mirai’ to be used interchangeably with a ‘promise’, allowing +side-effects to be performed upon resolution of a ‘mirai’; or + +[`crew`](https://wlandau.github.io/crew/) provides an interface that +makes it easy to deploy `mirai` for +[`shiny`](https://cran.r-project.org/package=shiny). The package +provides a [Shiny +vignette](https://wlandau.github.io/crew/articles/shiny.html) with +tutorial and sample code for this purpose. ### Thanks diff --git a/man/daemons.Rd b/man/daemons.Rd index 37a0310b9..141088274 100644 --- a/man/daemons.Rd +++ b/man/daemons.Rd @@ -55,8 +55,37 @@ and [ii] the associated private key.} is encrypted with a password) For security, should be provided through a function that returns this value, rather than directly.} -\item{...}{additional arguments passed through to \code{\link{dispatcher}} if -using dispatcher and/or \code{\link{daemon}} if launching local daemons.} +\item{...}{ + Arguments passed on to \code{\link[=dispatcher]{dispatcher}}, \code{\link[=daemon]{daemon}} + \describe{ + \item{\code{token}}{[default FALSE] if TRUE, appends a unique 40-character token +to each URL path the dispatcher listens at (not applicable for TCP URLs +which do not accept a path).} + \item{\code{lock}}{[default FALSE] if TRUE, sockets lock once a connection has been +accepted, preventing further connection attempts. This provides safety +against more than one daemon attempting to connect to a unique URL.} + \item{\code{output}}{[default FALSE] logical value, to output generated stdout / +stderr if TRUE, or else discard if FALSE. Specify as TRUE in the '...' +argument to \code{\link{daemons}} or \code{\link{launch_local}} to +provide redirection of output to the host process. Applicable only when +not using dispatcher.} + \item{\code{maxtasks}}{[default Inf] the maximum number of tasks to execute (task +limit) before exiting.} + \item{\code{idletime}}{[default Inf] maximum idle time, since completion of the last +task (in milliseconds) before exiting.} + \item{\code{walltime}}{[default Inf] soft walltime, or the minimum amount of real +time taken (in milliseconds) before exiting.} + \item{\code{timerstart}}{[default 0L] number of completed tasks after which to start +the timer for 'idletime' and 'walltime'. 0L implies timers are started +upon launch.} + \item{\code{cleanup}}{[default 7L] Integer additive bitmask controlling whether to +perform cleanup of the global environment (1L), reset loaded packages to +an initial state (2L), reset options to an initial state (4L), and +perform garbage collection (8L) after each evaluation. This option should +not normally be modified. Do not set unless you are certain you require +persistence across evaluations. Note: it may be an error to reset options +but not loaded packages if packages set options on load.} + }} \item{.compute}{[default 'default'] character compute profile to use for creating the daemons (each compute profile has its own set of daemons for diff --git a/man/dispatcher.Rd b/man/dispatcher.Rd index 9037de68a..5367ff264 100644 --- a/man/dispatcher.Rd +++ b/man/dispatcher.Rd @@ -61,8 +61,26 @@ certificate chain) and [ii] the associated private key.} is encrypted with a password) For security, should be provided through a function that returns this value, rather than directly.} -\item{...}{additional arguments passed through to \code{\link{daemon}} if -launching local daemons i.e. 'url' is not specified.} +\item{...}{ + Arguments passed on to \code{\link[=daemon]{daemon}} + \describe{ + \item{\code{maxtasks}}{[default Inf] the maximum number of tasks to execute (task +limit) before exiting.} + \item{\code{idletime}}{[default Inf] maximum idle time, since completion of the last +task (in milliseconds) before exiting.} + \item{\code{walltime}}{[default Inf] soft walltime, or the minimum amount of real +time taken (in milliseconds) before exiting.} + \item{\code{timerstart}}{[default 0L] number of completed tasks after which to start +the timer for 'idletime' and 'walltime'. 0L implies timers are started +upon launch.} + \item{\code{cleanup}}{[default 7L] Integer additive bitmask controlling whether to +perform cleanup of the global environment (1L), reset loaded packages to +an initial state (2L), reset options to an initial state (4L), and +perform garbage collection (8L) after each evaluation. This option should +not normally be modified. Do not set unless you are certain you require +persistence across evaluations. Note: it may be an error to reset options +but not loaded packages if packages set options on load.} + }} \item{monitor}{(for package internal use only) do not set this parameter.} diff --git a/tests/parallel/parallel-tests.R b/tests/parallel/parallel-tests.R index 9dc334b44..b98c85467 100644 --- a/tests/parallel/parallel-tests.R +++ b/tests/parallel/parallel-tests.R @@ -1,3 +1,7 @@ +rversion <- .subset2(getRversion(), 1L) +(rversion[1L] >= 4 && rversion[2L] >= 4 || rversion[1L] >= 5) || + stop("Requirement R >= 4.4 not met, stopped\n") + library(mirai) library(parallel) diff --git a/vignettes/.gitignore b/vignettes/.gitignore new file mode 100644 index 000000000..097b24163 --- /dev/null +++ b/vignettes/.gitignore @@ -0,0 +1,2 @@ +*.html +*.R diff --git a/vignettes/reference.Rmd b/vignettes/reference.Rmd new file mode 100644 index 000000000..6447764de --- /dev/null +++ b/vignettes/reference.Rmd @@ -0,0 +1,569 @@ +--- +title: "mirai - Reference Manual" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{mirai - Reference Manual} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + + + +### Table of Contents + +mirai logo + +1. [Example 1: Compute-intensive Operations](#example-1-compute-intensive-operations) +2. [Example 2: I/O-bound Operations](#example-2-io-bound-operations) +3. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) +4. [Daemons: Local Persistent Processes](#daemons-local-persistent-processes) +5. [Distributed Computing: Remote Daemons](#distributed-computing-remote-daemons) +6. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) +7. [Compute Profiles](#compute-profiles) +8. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) + + +### Example 1: Compute-intensive Operations + +Use case: minimise execution times by performing long-running tasks concurrently in separate processes. + +Multiple long computes (model fits etc.) can be performed in parallel on available computing cores. + +Use `mirai()` to evaluate an expression asynchronously in a separate, clean R process. + +A 'mirai' object is returned immediately. + + +```r +library(mirai) + +m <- mirai( + { + res <- rnorm(n) + m + res / rev(res) + }, + m = runif(1), + n = 1e8 +) + +m +#> < mirai > +#> - $data for evaluated result +``` + +Above, all specified `name = value` pairs are passed through to the 'mirai'. + +The 'mirai' yields an 'unresolved' logical NA whilst the async operation is ongoing. + + +```r +m$data +#> 'unresolved' logi NA +``` + + +Upon completion, the 'mirai' resolves automatically to the evaluated result. + + +```r +m$data |> str() +#> num [1:100000000] 2.828 2.788 0.593 -8.116 0.148 ... +``` + +Alternatively, explicitly call and wait for the result using `call_mirai()`. + + +```r +call_mirai(m)$data |> str() +#> num [1:100000000] 2.828 2.788 0.593 -8.116 0.148 ... +``` + +For easy programmatic use of `mirai()`, '.expr' accepts a pre-constructed language object, and also a list of named arguments passed via '.args'. So, the following would be equivalent to the above: + + +```r +expr <- quote({ + res <- rnorm(n) + m + res / rev(res) +}) + +args <- list(m = runif(1), n = 1e8) + +m <- mirai(.expr = expr, .args = args) + +call_mirai(m)$data |> str() +#> num [1:100000000] 0.822 -5.069 0.14 5.627 0.919 ... +``` + +[« Back to ToC](#table-of-contents) + +### Example 2: I/O-bound Operations + +Use case: ensure execution flow of the main process is not blocked. + +High-frequency real-time data cannot be written to file/database synchronously without disrupting the execution flow. + +Cache data in memory and use `mirai()` to perform periodic write operations concurrently in a separate process. + +Below, '.args' is used to pass a list of objects already present in the calling environment to the mirai by name. This is an alternative use of '.args', and may be combined with `...` to also pass in `name = value` pairs. + + +```r +library(mirai) + +x <- rnorm(1e6) +file <- tempfile() + +m <- mirai(write.csv(x, file = file), .args = list(x, file)) +``` + +A 'mirai' object is returned immediately. + +`unresolved()` may be used in control flow statements to perform actions which depend on resolution of the 'mirai', both before and after. + +This means there is no need to actually wait (block) for a 'mirai' to resolve, as the example below demonstrates. + + +```r +# unresolved() queries for resolution itself so no need to use it again within the while loop + +while (unresolved(m)) { + cat("while unresolved\n") + Sys.sleep(0.5) +} +#> while unresolved +#> while unresolved + +cat("Write complete:", is.null(m$data)) +#> Write complete: TRUE +``` + +Now actions which depend on the resolution may be processed, for example the next write. + +[« Back to ToC](#table-of-contents) + +### Example 3: Resilient Pipelines + +Use case: isolating code that can potentially fail in a separate process to ensure continued uptime. + +As part of a data science / machine learning pipeline, iterations of model training may periodically fail for stochastic and uncontrollable reasons (e.g. buggy memory management on graphics cards). + +Running each iteration in a 'mirai' isolates this potentially-problematic code such that even if it does fail, it does not bring down the entire pipeline. + + +```r +library(mirai) + +run_iteration <- function(i) { + + if (runif(1) < 0.1) stop("random error\n", call. = FALSE) # simulates a stochastic error rate + sprintf("iteration %d successful\n", i) + +} + +for (i in 1:10) { + + m <- mirai(run_iteration(i), .args = list(run_iteration, i)) + while (is_error_value(call_mirai(m)$data)) { + cat(m$data) + m <- mirai(run_iteration(i), .args = list(run_iteration, i)) + } + cat(m$data) + +} +#> iteration 1 successful +#> iteration 2 successful +#> iteration 3 successful +#> iteration 4 successful +#> iteration 5 successful +#> Error: random error +#> Error: random error +#> iteration 6 successful +#> iteration 7 successful +#> Error: random error +#> iteration 8 successful +#> iteration 9 successful +#> iteration 10 successful +``` + +Further, by testing the return value of each 'mirai' for errors, error-handling code is then able to automate recovery and re-attempts, as in the above example. Further details on [error handling](#errors-interrupts-and-timeouts) can be found in the section below. + +The end result is a resilient and fault-tolerant pipeline that minimises downtime by eliminating interruptions of long computes. + +[« Back to ToC](#table-of-contents) + +### Daemons: Local Persistent Processes + +Daemons, or persistent background processes, may be set to receive 'mirai' requests. + +This is potentially more efficient as new processes no longer need to be created on an *ad hoc* basis. + +#### With Dispatcher (default) + +Call `daemons()` specifying the number of daemons to launch. + + +```r +daemons(6) +#> [1] 6 +``` + + + +To view the current status, `status()` provides the number of active connections along with a matrix of statistics for each daemon. + + +```r +status() +#> $connections +#> [1] 1 +#> +#> $daemons +#> i online instance assigned complete +#> abstract://240ed790a962968bd2c045a5 1 1 1 0 0 +#> abstract://711ff183ce771fe3a3b712c2 2 1 1 0 0 +#> abstract://85592527215b7f538eb04f9e 3 1 1 0 0 +#> abstract://b8c4e8e778c0dfd3d957bb85 4 1 1 0 0 +#> abstract://fed5b52677897016bfa1b9c3 5 1 1 0 0 +#> abstract://84f2068e0fe8bb108ba55b75 6 1 1 0 0 +``` + +The default `dispatcher = TRUE` creates a `dispatcher()` background process that connects to individual daemon processes on the local machine. This ensures that tasks are dispatched efficiently on a first-in first-out (FIFO) basis to daemons for processing. Tasks are queued at the dispatcher and sent to a daemon as soon as it can accept the task for immediate execution. + +Dispatcher uses synchronisation primitives from [`nanonext`](https://doi.org/10.5281/zenodo.7903429), waiting upon rather than polling for tasks, which is efficient both in terms of consuming no resources while waiting, and also being fully synchronised with events (having no latency). + + +```r +daemons(0) +#> [1] 0 +``` + +Set the number of daemons to zero to reset. This reverts to the default of creating a new background process for each 'mirai' request. + +#### Without Dispatcher + +Alternatively, specifying `dispatcher = FALSE`, the background daemons connect directly to the host process. + + +```r +daemons(6, dispatcher = FALSE) +#> [1] 6 +``` + + +Requesting the status now shows 6 connections, along with the host URL at `$daemons`. + + +```r +status() +#> $connections +#> [1] 6 +#> +#> $daemons +#> [1] "abstract://c15fc79a6ab9ebf1817408ee" +``` + +This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons remain idle. + +The advantage of this approach is that it is low-level and does not require an additional dispatcher process. It is well-suited to working with similar-length tasks, or where the number of concurrent tasks typically does not exceed available daemons. + + +```r +daemons(0) +#> [1] 0 +``` + +Set the number of daemons to zero to reset. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: Remote Daemons + +The daemons interface may also be used to send tasks for computation to remote daemon processes on the network. + +Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections. + +The examples below use an illustrative local network IP address of '10.75.32.74'. + +A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below. + +IPv6 addresses are also supported and must be enclosed in square brackets `[]` to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would be specified as `tcp://[::ffff:a6f:50d]:5555`. + +#### Connecting to Remote Daemons Through Dispatcher + +The default `dispatcher = TRUE` creates a background `dispatcher()` process on the local machine, which listens to a vector of URLs that remote `daemon()` processes dial in to, with each daemon having its own unique URL. + +It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port. + + +```r +daemons(n = 4, url = "ws://10.75.32.74:5555") +#> [1] 4 +``` + +Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs. + +Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.: + + +```r +daemons(url = c("ws://10.75.32.74:5566/cpu", "ws://10.75.32.74:5566/gpu", "ws://10.75.32.74:7788/1")) +``` + +Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied. + +-- + +On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at: + +``` +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/1")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/2")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/3")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/4")' + +``` + +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. + +`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: + + +```r +launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.32.100", .)) +``` + +``` +#> [1] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/1',rs=c(10407,2132849204,678385573,1614909474,-118217413,-425668800,-1467206079))\"" +#> [2] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/2',rs=c(10407,-338879520,531920121,1404557287,1049906836,836803767,823372905))\"" +#> [3] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/3',rs=c(10407,-2114564378,1670187812,905031098,1374005312,1044737411,1978463798))\"" +#> [4] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/4',rs=c(10407,-251416338,906294615,-1357182089,-1619967674,-53885628,-1500505307))\"" +``` + +The returned vector comprises the shell commands executed on the remote machine. + +-- + +Requesting status, on the host machine: + + +```r +status() +#> $connections +#> [1] 1 +#> +#> $daemons +#> i online instance assigned complete +#> ws://10.75.32.74:5555/1 1 1 1 0 0 +#> ws://10.75.32.74:5555/2 2 1 1 0 0 +#> ws://10.75.32.74:5555/3 3 1 1 0 0 +#> ws://10.75.32.74:5555/4 4 1 1 0 0 +``` + +As per the local case, `$connections` shows the single connection to dispatcher, however `$daemons` now provides a matrix of statistics for the remote daemons. + +- `i` index number. +- `online` shows as 1 when there is an active connection, or else 0 if a daemon has yet to connect or has disconnected. +- `instance` increments by 1 every time there is a new connection at a URL. This counter is designed to track new daemon instances connecting after previous ones have ended (due to time-outs etc.). The count becomes negative immediately after a URL is regenerated by `saisei()`, but increments again once a new daemon connects. +- `assigned` shows the cumulative number of tasks assigned to the daemon. +- `complete` shows the cumulative number of tasks completed by the daemon. + +Dispatcher automatically adjusts to the number of daemons actually connected. Hence it is possible to dynamically scale up or down the number of daemons according to requirements (limited to the 'n' URLs assigned). + +To reset all connections and revert to default behaviour: + + +```r +daemons(0) +#> [1] 0 +``` + +Closing the connection causes the dispatcher to exit automatically, and in turn all connected daemons when their respective connections with the dispatcher are terminated. + +#### Connecting to Remote Daemons Directly + +By specifying `dispatcher = FALSE`, remote daemons connect directly to the host process. The host listens at a single URL, and distributes tasks to all connected daemons. + + +```r +daemons(url = "tcp://10.75.32.74:0", dispatcher = FALSE) +``` + +Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example: + + +```r +daemons(url = "tcp://:0", dispatcher = FALSE) +#> [1] "tcp://:34269" +``` + +Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. + +-- + +On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks: + +``` +Rscript -e 'mirai::daemon("tcp://10.75.32.74:34269")' +``` +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. + +`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: + + +```r +launch_remote("tcp://10.75.32.74:34269", command = "ssh", args = c("-p 22 10.75.32.100", .)) +``` + +``` +#> [1] "Rscript -e \"mirai::daemon('tcp://10.75.32.74:34269',rs=c(10407,11609875,-1276427976,1189810649,-872525882,-1804013681,-124708732))\"" +``` + +The returned vector comprises the shell commands executed on the remote machine. + +-- + +The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. + +`$connections` will show the actual number of connected daemons. + + +```r +status() +#> $connections +#> [1] 1 +#> +#> $daemons +#> [1] "tcp://:34269" +``` + +To reset all connections and revert to default behaviour: + + +```r +daemons(0) +#> [1] 0 +``` + +This causes all connected daemons to exit automatically. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: TLS Secure Connections + +TLS is available as an option to secure communications from the local machine to remote daemons. + +#### Zero-configuration + +An automatic zero-configuration default is implemented. Simply specify a secure URL of the form `wss://` or `tls+tcp://` when setting daemons. For example, on the IPv6 loopback address: + + +```r +daemons(n = 4, url = "wss://[::1]:5555") +#> [1] 4 +``` + +Single-use keys and certificates are automatically generated and configured, without requiring any further intervention. The private key is always retained on the host machine and never transmitted. + +The generated self-signed certificate is available via `launch_remote()`. This function conveniently constructs the full shell command to launch a daemon, including the correctly specified 'tls' argument to `daemon()`. + + +```r +launch_remote(1) +#> [1] "Rscript -e \"mirai::daemon('wss://[::1]:5555/1',tls=c('-----BEGIN CERTIFICATE-----\nMIIFLTCCAxWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMQwwCgYDVQQDDAM6OjEx\nETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAxMDEwMDAwMDBa\nFw0zMDEyMzEyMzU5NTlaMC4xDDAKBgNVBAMMAzo6MTERMA8GA1UECgwITmFub25l\neHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA\nvY+rtt09EZ2/mvnbjUUqpGh2LiQp5NCchNfVe9SO/4WzQqp2GNheZIVzg+n1qo2w\nLGPWuYSzP7anwp5CYxDH3kA358x0iuXPHourMdexMZD2RWzWR4WbJAZJaYLARrSH\nPPQTFUerR3XsBStZjnf+mrXAub0OpFTK9m/FB+Wa2IolTILd1Dr+Dra1SVrVYQPA\n57dmOFPSct9eUCcRaXpDRELFot6USApU/Q44CRW/FvUXFnuFxaqYAqKEI+acyTEa\nys9giRUhlIfS2Y5YcoxgsJy4Uuwk4PHqStj+0SR4/2B26RUq2T4ZXUVyHn4DscSi\nHK/ILUwgQhl9Y4i6IvqltoBe0PFWRabwz3u6XY8jT/4TSYETV9AR6U5qItL/3u0q\nYp/X8gqEh186Igl5FaCa2tEboHqbSNdDDJQF6eHdgvKjG9/thnMv/sZf8A3CALMA\npcT1KVENSAUaje/W0+bBlnDW2hHdultW5YcRpOIOitazyNHwigpZtX4iaM8kLZHa\nm6S40C/EVi8vug1nripb28dZe5TNa/jNHka1PPNNu5tjnUQ16X+hDqTVlWzb5Kh8\nvNhszDT9pqFdLPrWM/oHGlAAwld1QRfnXZtDp2XuxfZAsOBkEc6vaK4zDBEaGY1Z\n6vOLCdSTTYPiCYVnmhtCbIx0ZPBxEkbHgyeCmg5I/PUCAwEAAaNWMFQwEgYDVR0T\nAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUca42iRy3OeiOiuO6PtIB18QTFZkwHwYD\nVR0jBBgwFoAUca42iRy3OeiOiuO6PtIB18QTFZkwDQYJKoZIhvcNAQELBQADggIB\nAJvUOTO/FVd9xiLUEP7WcAzsJdCTeMx+A88LlAwGLpAPA+gYdeJgnn8sKtqx6IPe\n/jnBT9t/2nlPQVyL/674R9TLKynACbsRwDSLcDI9awB0NSiEErZ1VBgNqkOHIcgv\njOtvk7KI1407CEmaYC52W9yhoHFxSeBN7fK1eQx4PGv4+YxDA6shxuInNZrBFzEb\ndLLWV+vkWpY8aDKz9t9mgTpJYxJNTRmBLQ+3y/rSOBFFAxXNm/m/esKTAaWZbred\nzeGj0hzsehSkJ+oVCVQW4QL/jmfJMLtl0z+r9Kd175XJsDo/FY0BRr46OA/JO5Kg\nelJU+Cv4GuN+vtoy7SrewFRcMHELa23MjbhMKqRVc/oHUv5DM5ccF/eIJWgRanz/\nrd5BXa8ouQM3/Xgx24mz8NgcXC8dOCS4exb9POLhBN1keanuyJPw9iPZ+mS0Bo3H\nMB3uB6/aTW7E7KS+dhzxXfkchwVjhaW10YgrCD1YrSEXUgI7fe0WXJpD1EEL9hZ9\n53eqgtTF5cX6lw+DefpDCschKCV8jM9xq4RolhgGgvrHKss2kUgu9Xv+O0sTtyYr\nJb5ffcfRLkRXArToXQ+K1MqfqNyIoY0CkDpZbQ8vpF7ioGPQ6Tckm3G8nYkhGr4w\nVkhRgqvd+yehkyM5ZgYUBl+BXH7/pCl+XjhQuAwvX3Rg\n-----END CERTIFICATE-----\n',''),rs=c(10407,1692947812,-272206571,-1963797806,1811512363,-1639867024,-1392917839))\"" +``` + +The return value may be deployed manually on a remote machine by unescaping the double quotes around the call to `"mirai::daemon()"`, or directly via SSH or a resource manager by additionally specifying 'command' and 'args' to `launch_remote()`. + + + +#### CA Signed Certificates + +As an alternative to the zero-configuration option, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to your organisation. + +1. Generate a private key and CSR. The following resources describe how to do so: + +- using Mbed TLS: +- using OpenSSL: (Chapter 1.2 Key and Certificate Management) + +2. Send or provide the generated CSR to the CA for it to sign a new TLS certificate. + +- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, your TLS library should usually provide conversion utilities. +- Check also that your private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. + +3. When setting daemons, the TLS certificate and private key should be provided to the 'tls' argument of `daemons()`. + +- If the certificate and private key have been imported as character strings `cert` and `key` respectively, then the 'tls' argument may be specified as the character vector `c(cert, key)`. +- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this new file may be provided to the 'tls' argument. + +4. When launching daemons, the certificate chain to the CA should be supplied to the 'tls' argument of `daemon()` or `launch_remote()`. + +- The certificate chain should comprise multiple certificates, each between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers. The first one should be the newly-generated TLS certificate, the same supplied to `daemons()`, and the final one should be a CA root certificate. +- These are the only certificates required if your certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with your TLS certificate and ends with the certificate of the CA. +- If these are concatenated together as a single character string `certchain` (and assuming no certificate revocation list), then the character vector `c(certchain, "")` may be supplied to the relevant 'tls' argument. +- Alternatively, if these are written to a file (and the file replicated on the remote machines), then the 'tls' argument may also be specified as a path/filename (assuming these are the same on each machine). + +[« Back to ToC](#table-of-contents) + +### Compute Profiles + +The `daemons()` interface also allows the specification of compute profiles for managing tasks with heterogeneous compute requirements: + +- send tasks to different daemons or clusters of daemons with the appropriate specifications (in terms of CPUs / memory / GPU / accelerators etc.) +- split tasks between local and remote computation + +Simply specify the argument `.compute` when calling `daemons()` with a profile name (which is 'default' for the default profile). The daemons settings are saved under the named profile. + +To create a 'mirai' task using a specific compute profile, specify the '.compute' argument to `mirai()`, which defaults to the 'default' compute profile. + +Similarly, functions such as `status()`, `launch_local()` or `launch_remote()` should be specified with the desired '.compute' argument. + +[« Back to ToC](#table-of-contents) + +### Errors, Interrupts and Timeouts + +If execution in a mirai fails, the error message is returned as a character string of class 'miraiError' and 'errorValue' to facilitate debugging. `is_mirai_error()` may be used to test for mirai execution errors. + + +```r +m1 <- mirai(stop("occurred with a custom message", call. = FALSE)) +call_mirai(m1)$data +#> 'miraiError' chr Error: occurred with a custom message + +m2 <- mirai(mirai::mirai()) +call_mirai(m2)$data +#> 'miraiError' chr Error in mirai::mirai(): missing expression, perhaps wrap in {}? + +is_mirai_error(m2$data) +#> [1] TRUE +is_error_value(m2$data) +#> [1] TRUE +``` + +If a daemon instance is sent a user interrupt, the mirai will resolve to an empty character string of class 'miraiInterrupt' and 'errorValue'. `is_mirai_interrupt()` may be used to test for such interrupts. + + +```r +is_mirai_interrupt(m2$data) +#> [1] FALSE +``` + +If execution of a mirai surpasses the timeout set via the '.timeout' argument, the mirai will resolve to an 'errorValue'. This can, amongst other things, guard against mirai processes that have the potential to hang and never return. + + +```r +m3 <- mirai(nanonext::msleep(1000), .timeout = 500) +call_mirai(m3)$data +#> 'errorValue' int 5 | Timed out + +is_mirai_error(m3$data) +#> [1] FALSE +is_mirai_interrupt(m3$data) +#> [1] FALSE +is_error_value(m3$data) +#> [1] TRUE +``` + +`is_error_value()` tests for all mirai execution errors, user interrupts and timeouts. + +[« Back to ToC](#table-of-contents) diff --git a/vignettes/reference.Rmd.orig b/vignettes/reference.Rmd.orig new file mode 100644 index 000000000..8e62fc418 --- /dev/null +++ b/vignettes/reference.Rmd.orig @@ -0,0 +1,480 @@ +--- +title: "mirai - Reference Manual" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{mirai - Reference Manual} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + +```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>", + out.width = "100%" +) +``` + +### Table of Contents + +mirai logo + +1. [Example 1: Compute-intensive Operations](#example-1-compute-intensive-operations) +2. [Example 2: I/O-bound Operations](#example-2-io-bound-operations) +3. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) +4. [Daemons: Local Persistent Processes](#daemons-local-persistent-processes) +5. [Distributed Computing: Remote Daemons](#distributed-computing-remote-daemons) +6. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) +7. [Compute Profiles](#compute-profiles) +8. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) + + +### Example 1: Compute-intensive Operations + +Use case: minimise execution times by performing long-running tasks concurrently in separate processes. + +Multiple long computes (model fits etc.) can be performed in parallel on available computing cores. + +Use `mirai()` to evaluate an expression asynchronously in a separate, clean R process. + +A 'mirai' object is returned immediately. + +```{r exec} +library(mirai) + +m <- mirai( + { + res <- rnorm(n) + m + res / rev(res) + }, + m = runif(1), + n = 1e8 +) + +m +``` + +Above, all specified `name = value` pairs are passed through to the 'mirai'. + +The 'mirai' yields an 'unresolved' logical NA whilst the async operation is ongoing. + +```{r do} +m$data +``` +```{r dowhile, echo=FALSE} +call_mirai(m) +``` + +Upon completion, the 'mirai' resolves automatically to the evaluated result. + +```{r resolv} +m$data |> str() +``` + +Alternatively, explicitly call and wait for the result using `call_mirai()`. + +```{r call} +call_mirai(m)$data |> str() +``` + +For easy programmatic use of `mirai()`, '.expr' accepts a pre-constructed language object, and also a list of named arguments passed via '.args'. So, the following would be equivalent to the above: + +```{r equiv} +expr <- quote({ + res <- rnorm(n) + m + res / rev(res) +}) + +args <- list(m = runif(1), n = 1e8) + +m <- mirai(.expr = expr, .args = args) + +call_mirai(m)$data |> str() +``` + +[« Back to ToC](#table-of-contents) + +### Example 2: I/O-bound Operations + +Use case: ensure execution flow of the main process is not blocked. + +High-frequency real-time data cannot be written to file/database synchronously without disrupting the execution flow. + +Cache data in memory and use `mirai()` to perform periodic write operations concurrently in a separate process. + +Below, '.args' is used to pass a list of objects already present in the calling environment to the mirai by name. This is an alternative use of '.args', and may be combined with `...` to also pass in `name = value` pairs. + +```{r exec2} +library(mirai) + +x <- rnorm(1e6) +file <- tempfile() + +m <- mirai(write.csv(x, file = file), .args = list(x, file)) +``` + +A 'mirai' object is returned immediately. + +`unresolved()` may be used in control flow statements to perform actions which depend on resolution of the 'mirai', both before and after. + +This means there is no need to actually wait (block) for a 'mirai' to resolve, as the example below demonstrates. + +```{r call2} +# unresolved() queries for resolution itself so no need to use it again within the while loop + +while (unresolved(m)) { + cat("while unresolved\n") + Sys.sleep(0.5) +} + +cat("Write complete:", is.null(m$data)) + +``` + +Now actions which depend on the resolution may be processed, for example the next write. + +[« Back to ToC](#table-of-contents) + +### Example 3: Resilient Pipelines + +Use case: isolating code that can potentially fail in a separate process to ensure continued uptime. + +As part of a data science / machine learning pipeline, iterations of model training may periodically fail for stochastic and uncontrollable reasons (e.g. buggy memory management on graphics cards). + +Running each iteration in a 'mirai' isolates this potentially-problematic code such that even if it does fail, it does not bring down the entire pipeline. + +```{r exec3r} +library(mirai) + +run_iteration <- function(i) { + + if (runif(1) < 0.1) stop("random error\n", call. = FALSE) # simulates a stochastic error rate + sprintf("iteration %d successful\n", i) + +} + +for (i in 1:10) { + + m <- mirai(run_iteration(i), .args = list(run_iteration, i)) + while (is_error_value(call_mirai(m)$data)) { + cat(m$data) + m <- mirai(run_iteration(i), .args = list(run_iteration, i)) + } + cat(m$data) + +} + +``` + +Further, by testing the return value of each 'mirai' for errors, error-handling code is then able to automate recovery and re-attempts, as in the above example. Further details on [error handling](#errors-interrupts-and-timeouts) can be found in the section below. + +The end result is a resilient and fault-tolerant pipeline that minimises downtime by eliminating interruptions of long computes. + +[« Back to ToC](#table-of-contents) + +### Daemons: Local Persistent Processes + +Daemons, or persistent background processes, may be set to receive 'mirai' requests. + +This is potentially more efficient as new processes no longer need to be created on an *ad hoc* basis. + +#### With Dispatcher (default) + +Call `daemons()` specifying the number of daemons to launch. + +```{r daemons} +daemons(6) +``` + +```{r daemons2, include=FALSE} +Sys.sleep(1) +``` + +To view the current status, `status()` provides the number of active connections along with a matrix of statistics for each daemon. + +```{r daemons3} +status() +``` + +The default `dispatcher = TRUE` creates a `dispatcher()` background process that connects to individual daemon processes on the local machine. This ensures that tasks are dispatched efficiently on a first-in first-out (FIFO) basis to daemons for processing. Tasks are queued at the dispatcher and sent to a daemon as soon as it can accept the task for immediate execution. + +Dispatcher uses synchronisation primitives from [`nanonext`](https://doi.org/10.5281/zenodo.7903429), waiting upon rather than polling for tasks, which is efficient both in terms of consuming no resources while waiting, and also being fully synchronised with events (having no latency). + +```{r daemons4} +daemons(0) +``` + +Set the number of daemons to zero to reset. This reverts to the default of creating a new background process for each 'mirai' request. + +#### Without Dispatcher + +Alternatively, specifying `dispatcher = FALSE`, the background daemons connect directly to the host process. + +```{r daemonsq} +daemons(6, dispatcher = FALSE) +``` +```{r daemonsq2, include=FALSE} +Sys.sleep(0.5) +``` + +Requesting the status now shows 6 connections, along with the host URL at `$daemons`. + +```{r daemonsqv} +status() +``` + +This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons remain idle. + +The advantage of this approach is that it is low-level and does not require an additional dispatcher process. It is well-suited to working with similar-length tasks, or where the number of concurrent tasks typically does not exceed available daemons. + +```{r daemons5} +daemons(0) +``` + +Set the number of daemons to zero to reset. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: Remote Daemons + +The daemons interface may also be used to send tasks for computation to remote daemon processes on the network. + +Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections. + +The examples below use an illustrative local network IP address of '10.75.32.74'. + +A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below. + +IPv6 addresses are also supported and must be enclosed in square brackets `[]` to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would be specified as `tcp://[::ffff:a6f:50d]:5555`. + +#### Connecting to Remote Daemons Through Dispatcher + +The default `dispatcher = TRUE` creates a background `dispatcher()` process on the local machine, which listens to a vector of URLs that remote `daemon()` processes dial in to, with each daemon having its own unique URL. + +It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port. + +```{r localqueue} +daemons(n = 4, url = "ws://10.75.32.74:5555") +``` + +Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs. + +Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.: + +```{r vectorqueue, eval=FALSE} +daemons(url = c("ws://10.75.32.74:5566/cpu", "ws://10.75.32.74:5566/gpu", "ws://10.75.32.74:7788/1")) +``` + +Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied. + +-- + +On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at: + +``` +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/1")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/2")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/3")' +Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/4")' + +``` + +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. + +`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: + +```{r launchremote, eval=FALSE} +launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.32.100", .)) +``` +```{r launchremotereal, echo=FALSE} +launch_remote(1:4) +``` + +The returned vector comprises the shell commands executed on the remote machine. + +-- + +Requesting status, on the host machine: + +```{r remotev2} +status() +``` + +As per the local case, `$connections` shows the single connection to dispatcher, however `$daemons` now provides a matrix of statistics for the remote daemons. + +- `i` index number. +- `online` shows as 1 when there is an active connection, or else 0 if a daemon has yet to connect or has disconnected. +- `instance` increments by 1 every time there is a new connection at a URL. This counter is designed to track new daemon instances connecting after previous ones have ended (due to time-outs etc.). The count becomes negative immediately after a URL is regenerated by `saisei()`, but increments again once a new daemon connects. +- `assigned` shows the cumulative number of tasks assigned to the daemon. +- `complete` shows the cumulative number of tasks completed by the daemon. + +Dispatcher automatically adjusts to the number of daemons actually connected. Hence it is possible to dynamically scale up or down the number of daemons according to requirements (limited to the 'n' URLs assigned). + +To reset all connections and revert to default behaviour: + +```{r reset2} +daemons(0) +``` + +Closing the connection causes the dispatcher to exit automatically, and in turn all connected daemons when their respective connections with the dispatcher are terminated. + +#### Connecting to Remote Daemons Directly + +By specifying `dispatcher = FALSE`, remote daemons connect directly to the host process. The host listens at a single URL, and distributes tasks to all connected daemons. + +```{r remote, eval=FALSE} +daemons(url = "tcp://10.75.32.74:0", dispatcher = FALSE) +``` + +Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example: + +```{r remotealt} +daemons(url = "tcp://:0", dispatcher = FALSE) +``` + +Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. + +-- + +On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks: + +``` +Rscript -e 'mirai::daemon("tcp://10.75.32.74:0")' +``` +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. + +`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: + +```{r launchremoteu, eval=FALSE} +launch_remote("tcp://10.75.32.74:0", command = "ssh", args = c("-p 22 10.75.32.100", .)) +``` +```{r launchremoteureal, echo=FALSE} +launch_remote("tcp://10.75.32.74:0") +``` + +The returned vector comprises the shell commands executed on the remote machine. + +-- + +The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. + +`$connections` will show the actual number of connected daemons. + +```{r remotev} +status() +``` + +To reset all connections and revert to default behaviour: + +```{r reset} +daemons(0) +``` + +This causes all connected daemons to exit automatically. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: TLS Secure Connections + +TLS is available as an option to secure communications from the local machine to remote daemons. + +#### Zero-configuration + +An automatic zero-configuration default is implemented. Simply specify a secure URL of the form `wss://` or `tls+tcp://` when setting daemons. For example, on the IPv6 loopback address: + +```{r tlsremote} +daemons(n = 4, url = "wss://[::1]:5555") +``` + +Single-use keys and certificates are automatically generated and configured, without requiring any further intervention. The private key is always retained on the host machine and never transmitted. + +The generated self-signed certificate is available via `launch_remote()`. This function conveniently constructs the full shell command to launch a daemon, including the correctly specified 'tls' argument to `daemon()`. + +```{r launch_remote} +launch_remote(1) +``` + +The return value may be deployed manually on a remote machine by unescaping the double quotes around the call to `"mirai::daemon()"`, or directly via SSH or a resource manager by additionally specifying 'command' and 'args' to `launch_remote()`. + +```{r tlsclose, include=FALSE} +daemons(0) +``` + +#### CA Signed Certificates + +As an alternative to the zero-configuration option, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to your organisation. + +1. Generate a private key and CSR. The following resources describe how to do so: + +- using Mbed TLS: +- using OpenSSL: (Chapter 1.2 Key and Certificate Management) + +2. Send or provide the generated CSR to the CA for it to sign a new TLS certificate. + +- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, your TLS library should usually provide conversion utilities. +- Check also that your private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. + +3. When setting daemons, the TLS certificate and private key should be provided to the 'tls' argument of `daemons()`. + +- If the certificate and private key have been imported as character strings `cert` and `key` respectively, then the 'tls' argument may be specified as the character vector `c(cert, key)`. +- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this new file may be provided to the 'tls' argument. + +4. When launching daemons, the certificate chain to the CA should be supplied to the 'tls' argument of `daemon()` or `launch_remote()`. + +- The certificate chain should comprise multiple certificates, each between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers. The first one should be the newly-generated TLS certificate, the same supplied to `daemons()`, and the final one should be a CA root certificate. +- These are the only certificates required if your certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with your TLS certificate and ends with the certificate of the CA. +- If these are concatenated together as a single character string `certchain` (and assuming no certificate revocation list), then the character vector `c(certchain, "")` may be supplied to the relevant 'tls' argument. +- Alternatively, if these are written to a file (and the file replicated on the remote machines), then the 'tls' argument may also be specified as a path/filename (assuming these are the same on each machine). + +[« Back to ToC](#table-of-contents) + +### Compute Profiles + +The `daemons()` interface also allows the specification of compute profiles for managing tasks with heterogeneous compute requirements: + +- send tasks to different daemons or clusters of daemons with the appropriate specifications (in terms of CPUs / memory / GPU / accelerators etc.) +- split tasks between local and remote computation + +Simply specify the argument `.compute` when calling `daemons()` with a profile name (which is 'default' for the default profile). The daemons settings are saved under the named profile. + +To create a 'mirai' task using a specific compute profile, specify the '.compute' argument to `mirai()`, which defaults to the 'default' compute profile. + +Similarly, functions such as `status()`, `launch_local()` or `launch_remote()` should be specified with the desired '.compute' argument. + +[« Back to ToC](#table-of-contents) + +### Errors, Interrupts and Timeouts + +If execution in a mirai fails, the error message is returned as a character string of class 'miraiError' and 'errorValue' to facilitate debugging. `is_mirai_error()` may be used to test for mirai execution errors. + +```{r errorexample} +m1 <- mirai(stop("occurred with a custom message", call. = FALSE)) +call_mirai(m1)$data + +m2 <- mirai(mirai::mirai()) +call_mirai(m2)$data + +is_mirai_error(m2$data) +is_error_value(m2$data) +``` + +If a daemon instance is sent a user interrupt, the mirai will resolve to an empty character string of class 'miraiInterrupt' and 'errorValue'. `is_mirai_interrupt()` may be used to test for such interrupts. + +```{r interruptexample} +is_mirai_interrupt(m2$data) +``` + +If execution of a mirai surpasses the timeout set via the '.timeout' argument, the mirai will resolve to an 'errorValue'. This can, amongst other things, guard against mirai processes that have the potential to hang and never return. + +```{r timeouts} +m3 <- mirai(nanonext::msleep(1000), .timeout = 500) +call_mirai(m3)$data + +is_mirai_error(m3$data) +is_mirai_interrupt(m3$data) +is_error_value(m3$data) +``` + +`is_error_value()` tests for all mirai execution errors, user interrupts and timeouts. + +[« Back to ToC](#table-of-contents)