From 56dc7d056927587f311bb645dfbe57cb361fcb83 Mon Sep 17 00:00:00 2001 From: shikokuchuo <53399081+shikokuchuo@users.noreply.github.com> Date: Thu, 5 Oct 2023 17:14:30 +0100 Subject: [PATCH] update reference vignette --- R/daemons.R | 4 +- man/daemons.Rd | 4 +- vignettes/precompile.R | 1 + vignettes/reference.Rmd | 304 +++++++++++++++++++++++------------ vignettes/reference.Rmd.orig | 195 +++++++++++++--------- 5 files changed, 323 insertions(+), 185 deletions(-) diff --git a/R/daemons.R b/R/daemons.R index 90f3cc8d4..edd8b6fc4 100644 --- a/R/daemons.R +++ b/R/daemons.R @@ -134,8 +134,8 @@ #' #' Specify 'remote' with a call to \code{\link{remote_config}} or #' \code{\link{ssh_config}} to launch daemons on remote machines. Otherwise, -#' \code{\link{launch_remote}} may also be used to generate the shell -#' commands to deploy daemons manually on remote resources. +#' \code{\link{launch_remote}} may be used to generate the shell commands to +#' deploy daemons manually on remote resources. #' #' IPv6 addresses are also supported and must be enclosed in square brackets #' [ ] to avoid confusion with the final colon separating the port. For diff --git a/man/daemons.Rd b/man/daemons.Rd index a6a36ae29..3a9e68c04 100644 --- a/man/daemons.Rd +++ b/man/daemons.Rd @@ -152,8 +152,8 @@ Use \code{daemons(0)} to reset daemon connections: Specify 'remote' with a call to \code{\link{remote_config}} or \code{\link{ssh_config}} to launch daemons on remote machines. Otherwise, - \code{\link{launch_remote}} may also be used to generate the shell - commands to deploy daemons manually on remote resources. + \code{\link{launch_remote}} may be used to generate the shell commands to + deploy daemons manually on remote resources. IPv6 addresses are also supported and must be enclosed in square brackets [ ] to avoid confusion with the final colon separating the port. For diff --git a/vignettes/precompile.R b/vignettes/precompile.R index 547445124..1387ab3e9 100644 --- a/vignettes/precompile.R +++ b/vignettes/precompile.R @@ -1,2 +1,3 @@ # Vignettes precompiled as require connecting to remote resources interactively +set.seed(5731462) knitr::knit("vignettes/reference.Rmd.orig", "vignettes/reference.Rmd") diff --git a/vignettes/reference.Rmd b/vignettes/reference.Rmd index 6447764de..48a7ae549 100644 --- a/vignettes/reference.Rmd +++ b/vignettes/reference.Rmd @@ -18,9 +18,11 @@ vignette: > 3. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) 4. [Daemons: Local Persistent Processes](#daemons-local-persistent-processes) 5. [Distributed Computing: Remote Daemons](#distributed-computing-remote-daemons) -6. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) -7. [Compute Profiles](#compute-profiles) -8. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) +6. [Distributed Computing: Launching Daemons](#distributed-computing-launching-daemons) +7. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) +8. [Compute Profiles](#compute-profiles) +9. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) +10. [Parallel Clusters](#parallel-clusters) ### Example 1: Compute-intensive Operations @@ -67,7 +69,7 @@ Upon completion, the 'mirai' resolves automatically to the evaluated result. ```r m$data |> str() -#> num [1:100000000] 2.828 2.788 0.593 -8.116 0.148 ... +#> num [1:100000000] 1.189 1.076 1.128 1.139 0.464 ... ``` Alternatively, explicitly call and wait for the result using `call_mirai()`. @@ -75,7 +77,7 @@ Alternatively, explicitly call and wait for the result using `call_mirai()`. ```r call_mirai(m)$data |> str() -#> num [1:100000000] 2.828 2.788 0.593 -8.116 0.148 ... +#> num [1:100000000] 1.189 1.076 1.128 1.139 0.464 ... ``` For easy programmatic use of `mirai()`, '.expr' accepts a pre-constructed language object, and also a list of named arguments passed via '.args'. So, the following would be equivalent to the above: @@ -92,7 +94,7 @@ args <- list(m = runif(1), n = 1e8) m <- mirai(.expr = expr, .args = args) call_mirai(m)$data |> str() -#> num [1:100000000] 0.822 -5.069 0.14 5.627 0.919 ... +#> num [1:100000000] 0.266 3.14 2.256 -1.264 -1.457 ... ``` [« Back to ToC](#table-of-contents) @@ -133,6 +135,7 @@ while (unresolved(m)) { } #> while unresolved #> while unresolved +#> while unresolved cat("Write complete:", is.null(m$data)) #> Write complete: TRUE @@ -172,16 +175,16 @@ for (i in 1:10) { } #> iteration 1 successful +#> Error: random error #> iteration 2 successful #> iteration 3 successful #> iteration 4 successful #> iteration 5 successful -#> Error: random error -#> Error: random error #> iteration 6 successful #> iteration 7 successful #> Error: random error #> iteration 8 successful +#> Error: random error #> iteration 9 successful #> iteration 10 successful ``` @@ -220,12 +223,12 @@ status() #> #> $daemons #> i online instance assigned complete -#> abstract://240ed790a962968bd2c045a5 1 1 1 0 0 -#> abstract://711ff183ce771fe3a3b712c2 2 1 1 0 0 -#> abstract://85592527215b7f538eb04f9e 3 1 1 0 0 -#> abstract://b8c4e8e778c0dfd3d957bb85 4 1 1 0 0 -#> abstract://fed5b52677897016bfa1b9c3 5 1 1 0 0 -#> abstract://84f2068e0fe8bb108ba55b75 6 1 1 0 0 +#> abstract://6312ccdb468419e06af2b3ca 1 1 1 0 0 +#> abstract://7c9e1c59db0733de4f1766b2 2 1 1 0 0 +#> abstract://517dd18bd52d390baab17179 3 1 1 0 0 +#> abstract://8b6f18aa03859c506c68dfc6 4 1 1 0 0 +#> abstract://af6dc146df02bd74f8cfa52a 5 1 1 0 0 +#> abstract://b2e1016f0faa78361ecfe022 6 1 1 0 0 ``` The default `dispatcher = TRUE` creates a `dispatcher()` background process that connects to individual daemon processes on the local machine. This ensures that tasks are dispatched efficiently on a first-in first-out (FIFO) basis to daemons for processing. Tasks are queued at the dispatcher and sent to a daemon as soon as it can accept the task for immediate execution. @@ -260,10 +263,10 @@ status() #> [1] 6 #> #> $daemons -#> [1] "abstract://c15fc79a6ab9ebf1817408ee" +#> [1] "abstract://40b43c93f0756cd7ffc5cf79" ``` -This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons remain idle. +This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons are idle having already completed their tasks. The advantage of this approach is that it is low-level and does not require an additional dispatcher process. It is well-suited to working with similar-length tasks, or where the number of concurrent tasks typically does not exceed available daemons. @@ -281,70 +284,34 @@ Set the number of daemons to zero to reset. The daemons interface may also be used to send tasks for computation to remote daemon processes on the network. -Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections. - -The examples below use an illustrative local network IP address of '10.75.32.74'. - -A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below. +Call `daemons()` specifying 'url' as a character string such as: 'tcp://10.90.127.129:5555' at which daemon processes should connect to. IPv6 addresses are also supported and must be enclosed in square brackets `[]` to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would be specified as `tcp://[::ffff:a6f:50d]:5555`. +For options on actually launching the daemons, please see the next section. + #### Connecting to Remote Daemons Through Dispatcher The default `dispatcher = TRUE` creates a background `dispatcher()` process on the local machine, which listens to a vector of URLs that remote `daemon()` processes dial in to, with each daemon having its own unique URL. It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port. - -```r -daemons(n = 4, url = "ws://10.75.32.74:5555") -#> [1] 4 -``` - -Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs. - -Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.: +Supplying a vector of URLs allows the use of arbitrary port numbers / paths. 'n' does not need to be specified if it can be inferred from the length of the 'url' vector, for example: ```r -daemons(url = c("ws://10.75.32.74:5566/cpu", "ws://10.75.32.74:5566/gpu", "ws://10.75.32.74:7788/1")) -``` - -Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied. - --- - -On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at: - +daemons(url = c("ws://10.90.127.129:5566/cpu", "ws://10.90.127.129:5566/gpu", "ws://10.90.127.129:7788/1")) ``` -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/1")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/2")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/3")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/4")' -``` - -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. - -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: +Alternatively, below a single URL is supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce the URLs. ```r -launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.32.100", .)) -``` - -``` -#> [1] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/1',rs=c(10407,2132849204,678385573,1614909474,-118217413,-425668800,-1467206079))\"" -#> [2] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/2',rs=c(10407,-338879520,531920121,1404557287,1049906836,836803767,823372905))\"" -#> [3] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/3',rs=c(10407,-2114564378,1670187812,905031098,1374005312,1044737411,1978463798))\"" -#> [4] "Rscript -e \"mirai::daemon('ws://10.75.32.74:5555/4',rs=c(10407,-251416338,906294615,-1357182089,-1619967674,-53885628,-1500505307))\"" +daemons(n = 4, url = "ws://10.90.127.129:5555") +#> [1] 4 ``` -The returned vector comprises the shell commands executed on the remote machine. - --- - -Requesting status, on the host machine: +Requesting status on the host machine: ```r @@ -353,11 +320,11 @@ status() #> [1] 1 #> #> $daemons -#> i online instance assigned complete -#> ws://10.75.32.74:5555/1 1 1 1 0 0 -#> ws://10.75.32.74:5555/2 2 1 1 0 0 -#> ws://10.75.32.74:5555/3 3 1 1 0 0 -#> ws://10.75.32.74:5555/4 4 1 1 0 0 +#> i online instance assigned complete +#> ws://10.90.127.129:5555/1 1 0 0 0 0 +#> ws://10.90.127.129:5555/2 2 0 0 0 0 +#> ws://10.90.127.129:5555/3 3 0 0 0 0 +#> ws://10.90.127.129:5555/4 4 0 0 0 0 ``` As per the local case, `$connections` shows the single connection to dispatcher, however `$daemons` now provides a matrix of statistics for the remote daemons. @@ -386,66 +353,110 @@ By specifying `dispatcher = FALSE`, remote daemons connect directly to the host ```r -daemons(url = "tcp://10.75.32.74:0", dispatcher = FALSE) +daemons(url = "tcp://10.90.127.129:0", dispatcher = FALSE) +#> [1] "tcp://10.90.127.129:42655" ``` -Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example: +Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. + +The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. + +`$connections` will show the actual number of connected daemons. ```r -daemons(url = "tcp://:0", dispatcher = FALSE) -#> [1] "tcp://:34269" +status() +#> $connections +#> [1] 0 +#> +#> $daemons +#> [1] "tcp://10.90.127.129:42655" ``` -Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. - --- +To reset all connections and revert to default behaviour: -On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks: +```r +daemons(0) +#> [1] 0 ``` -Rscript -e 'mirai::daemon("tcp://10.75.32.74:34269")' -``` -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: +This causes all connected daemons to exit automatically. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: Launching Daemons + +To launch remote daemons, supply a remote launch configuration to the 'remote' argument of `daemons()` when setting up daemons, or `launch_remote()` at any time afterwards. + +`ssh_config()` may be used to generate a remote launch configuration if there is SSH access to the remote machine, or else `remote_config()` provides a flexible method for generating a configuration involving a custom resource manager / application. + +#### SSH Direct Connection + +The first example below launches 4 daemons on the machine 10.75.32.90 (using the default SSH port of 22 as this was not specified), connecting back to the dispatcher URLs: ```r -launch_remote("tcp://10.75.32.74:34269", command = "ssh", args = c("-p 22 10.75.32.100", .)) +daemons( + n = 4, + url = "ws://10.75.32.74:5555", + remote = ssh_config(remotes = "ssh://10.75.32.90") +) ``` +The second example below launches one daemon on each of 10.75.32.90 and 10.75.32.91 using the custom SSH port of 222: + +```r +daemons( + n = 2, + url = "ws://10.75.32.74:5555", + remote = ssh_config(c("ssh://10.75.32.90:222", "ssh://10.75.32.91:222")) +) ``` -#> [1] "Rscript -e \"mirai::daemon('tcp://10.75.32.74:34269',rs=c(10407,11609875,-1276427976,1189810649,-872525882,-1804013681,-124708732))\"" -``` +In the above examples, as the remote daemons connect back directly, port 5555 on the local machine must be open to incoming connections from the remote addresses. -The returned vector comprises the shell commands executed on the remote machine. +#### SSH Tunnelling --- +Use of SSH tunnelling provides a convenient way to launch remote daemons without requiring the remote machine to be able to access the host. Often firewall configurations or security policies may prevent opening a port to accept outside connections. -The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. +In these cases SSH tunnelling offers a solution by creating a tunnel once the initial SSH connection is made. For simplicity, this SSH tunnelling implementation uses the same port on both the side of the host and that of the corresponding node. SSH key-based authentication must also already be in place. -`$connections` will show the actual number of connected daemons. +Tunnelling requires the hostname for 'url' specified when setting up daemons to be either 'localhost' or '127.0.0.1'. This is as the tunnel is created between localhost:port or equivalently 127.0.0.1:port on each machine. The host listens to its localhost:port and the remotes each dial into localhost:port on their own respective machines. + +The below example launches 2 nodes on the remote machine 10.75.32.90 using SSH tunnelling over port 5555 ('url' hostname is specified as 'localhost'): ```r -status() -#> $connections -#> [1] 1 -#> -#> $daemons -#> [1] "tcp://:34269" +daemons( + url = "tcp://localhost:5555", + remote = ssh_config( + remotes = c("ssh://10.75.32.90", "ssh://10.75.32.90"), + tunnel = TRUE + ) +) ``` -To reset all connections and revert to default behaviour: +#### Manual Deployment + +As an alternative to automated launches, calling `launch_remote()` without specifying 'remote' may be used to return the shell commands for deploying daemons manually. The printed return values may be copy / pasted directly to a remote machine. ```r +daemons(n = 2, url = "tcp://[::1]:0") +#> [1] 2 + +launch_remote(1:2) +#> [1] +#> Rscript -e "mirai::daemon('tcp://[::1]:34953',rs=c(10407,-1166731391,1500929870,99567479,783566156,-752068131,-1259262214))" +#> +#> [2] +#> Rscript -e "mirai::daemon('tcp://[::1]:43359',rs=c(10407,-1582707955,-850336355,-1200729574,699484280,-311519171,1433330543))" + daemons(0) #> [1] 0 ``` -This causes all connected daemons to exit automatically. +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying the argument `asyncdial = TRUE` will allow daemons to wait (indefinitely) for a connection to become available. [« Back to ToC](#table-of-contents) @@ -470,37 +481,71 @@ The generated self-signed certificate is available via `launch_remote()`. This f ```r launch_remote(1) -#> [1] "Rscript -e \"mirai::daemon('wss://[::1]:5555/1',tls=c('-----BEGIN CERTIFICATE-----\nMIIFLTCCAxWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMQwwCgYDVQQDDAM6OjEx\nETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAxMDEwMDAwMDBa\nFw0zMDEyMzEyMzU5NTlaMC4xDDAKBgNVBAMMAzo6MTERMA8GA1UECgwITmFub25l\neHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA\nvY+rtt09EZ2/mvnbjUUqpGh2LiQp5NCchNfVe9SO/4WzQqp2GNheZIVzg+n1qo2w\nLGPWuYSzP7anwp5CYxDH3kA358x0iuXPHourMdexMZD2RWzWR4WbJAZJaYLARrSH\nPPQTFUerR3XsBStZjnf+mrXAub0OpFTK9m/FB+Wa2IolTILd1Dr+Dra1SVrVYQPA\n57dmOFPSct9eUCcRaXpDRELFot6USApU/Q44CRW/FvUXFnuFxaqYAqKEI+acyTEa\nys9giRUhlIfS2Y5YcoxgsJy4Uuwk4PHqStj+0SR4/2B26RUq2T4ZXUVyHn4DscSi\nHK/ILUwgQhl9Y4i6IvqltoBe0PFWRabwz3u6XY8jT/4TSYETV9AR6U5qItL/3u0q\nYp/X8gqEh186Igl5FaCa2tEboHqbSNdDDJQF6eHdgvKjG9/thnMv/sZf8A3CALMA\npcT1KVENSAUaje/W0+bBlnDW2hHdultW5YcRpOIOitazyNHwigpZtX4iaM8kLZHa\nm6S40C/EVi8vug1nripb28dZe5TNa/jNHka1PPNNu5tjnUQ16X+hDqTVlWzb5Kh8\nvNhszDT9pqFdLPrWM/oHGlAAwld1QRfnXZtDp2XuxfZAsOBkEc6vaK4zDBEaGY1Z\n6vOLCdSTTYPiCYVnmhtCbIx0ZPBxEkbHgyeCmg5I/PUCAwEAAaNWMFQwEgYDVR0T\nAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUca42iRy3OeiOiuO6PtIB18QTFZkwHwYD\nVR0jBBgwFoAUca42iRy3OeiOiuO6PtIB18QTFZkwDQYJKoZIhvcNAQELBQADggIB\nAJvUOTO/FVd9xiLUEP7WcAzsJdCTeMx+A88LlAwGLpAPA+gYdeJgnn8sKtqx6IPe\n/jnBT9t/2nlPQVyL/674R9TLKynACbsRwDSLcDI9awB0NSiEErZ1VBgNqkOHIcgv\njOtvk7KI1407CEmaYC52W9yhoHFxSeBN7fK1eQx4PGv4+YxDA6shxuInNZrBFzEb\ndLLWV+vkWpY8aDKz9t9mgTpJYxJNTRmBLQ+3y/rSOBFFAxXNm/m/esKTAaWZbred\nzeGj0hzsehSkJ+oVCVQW4QL/jmfJMLtl0z+r9Kd175XJsDo/FY0BRr46OA/JO5Kg\nelJU+Cv4GuN+vtoy7SrewFRcMHELa23MjbhMKqRVc/oHUv5DM5ccF/eIJWgRanz/\nrd5BXa8ouQM3/Xgx24mz8NgcXC8dOCS4exb9POLhBN1keanuyJPw9iPZ+mS0Bo3H\nMB3uB6/aTW7E7KS+dhzxXfkchwVjhaW10YgrCD1YrSEXUgI7fe0WXJpD1EEL9hZ9\n53eqgtTF5cX6lw+DefpDCschKCV8jM9xq4RolhgGgvrHKss2kUgu9Xv+O0sTtyYr\nJb5ffcfRLkRXArToXQ+K1MqfqNyIoY0CkDpZbQ8vpF7ioGPQ6Tckm3G8nYkhGr4w\nVkhRgqvd+yehkyM5ZgYUBl+BXH7/pCl+XjhQuAwvX3Rg\n-----END CERTIFICATE-----\n',''),rs=c(10407,1692947812,-272206571,-1963797806,1811512363,-1639867024,-1392917839))\"" -``` +#> [1] +#> Rscript -e "mirai::daemon('wss://[::1]:5555/1',tls=c('-----BEGIN CERTIFICATE----- +#> MIIFLTCCAxWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMQwwCgYDVQQDDAM6OjEx +#> ETAPBgNVBAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDAeFw0wMTAxMDEwMDAwMDBa +#> Fw0zMDEyMzEyMzU5NTlaMC4xDDAKBgNVBAMMAzo6MTERMA8GA1UECgwITmFub25l +#> eHQxCzAJBgNVBAYTAkpQMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA +#> 2RQwZxKwP62oiIo1eLEpVuhDGS4etsS3BepmpztoWcEDoHZz4YlT3LCK5+o6EOZ/ +#> dXDTjphGNM98BUIzq+luy2sQHHOy0rdTFYBeT6Mzr8CF0wrFy+eUERwrKZJWoQWt +#> scU+6dI6VdMV2Qo7ySbDlL2xXGlT6hzQ9YtrGGVMME2r/G+Vym+clc7S3v65M1i7 +#> B0kppRW0LRmrPrVGPov7XNfaK1UhdNVnj1OqXIGfp+LsqqZuPl5BCDnZlWHOPL8O +#> 4M/XcbkQPIM+nJWy7mGcLouePJ6lbiDsBGW9f0LhV7NlIi7XvCxMSCxOu0+WXQW8 +#> 92oDiPF8PzAHzbtTDFV7b+Cngqzzpqn137XKmWEoI94WrZXFzT4lHdyXdD9ZyABm +#> AKKK+RXYfd0NxRmJNYr+tT0MaC5nIn0ZU9vz59Varz/3MXYL7EeeyvW9S6b/iCh0 +#> 2jU7FAqzWaU9Qf9CnYVs2/nXHdc5G4HzNMeoU9OU92W4Nkl+RxswS5Ggbgo4Wvao +#> pRmN8woEetodL2851bZwVelFse1FAQVKjpjJemNv5Nz1rEwEM9Uc3BSFWSgI/EwG +#> /m7eRYgNrFHItsN+2LpDnZrNbPpz5sLS6aTTfN181BmCmgzaRDO+k2ldT+ppRvUR +#> vZBFeoIG7NI/ktROuRVNDO7FnONYZfQN9P8sOBgEvM0CAwEAAaNWMFQwEgYDVR0T +#> AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUMxSi4qOuoG7isCHlZ/xrMPtvcj4wHwYD +#> VR0jBBgwFoAUMxSi4qOuoG7isCHlZ/xrMPtvcj4wDQYJKoZIhvcNAQELBQADggIB +#> AE9e0mrRgvY3iuAsgMJF0uhtgpgDUhpIKAb3k8Dz73hi5XPOVSd++cGTX8u2FwOZ +#> QikMa7WKMgT6NZKlYcpoULmRq5OkbtUo0ayvxhQfwLKtlS0n/Gk08y3slIRAyHiJ +#> 1GQV+CguMnCWBLTDzIt6PvGHzgKAgXldf1jULAC97Dp/OxiAKavdDwmytxhvuoOr +#> fckz8whYczDOSafYt7RyWDq2g93ieFVjzLBq7eqv+ivRIjRLC4zza6tTd+lI+6cl +#> Y6LFV14bVqmfc6mfWwTl/Op4u8XRrsnPYN5pDP/pmcQk2chMZ6Ann/dG6kHwFAr7 +#> xb9qcS2TbIVqrrsORRIkS8PWSNnbcqcurqYW0Xu8BxmBQqoeipXXnXBwZoI715D5 +#> S+YKyH8UEKIvPj56jEmhYRF5YeKlwZKkejQDMabKsMKXND23VYbDiZwLuLOHkpAB +#> QTXv8dcZ9bComdLdy62pUtt3PftfgbxlM1jlj6jDIlI+M4uPCAdVCMKHQkioP+gn +#> A2XQCXFaEp2jUJrE1gjV6R1Gy45qglZt3cHQievGp1wkGkOUOA6OhFMD/xb6EwG1 +#> 6TSij8QPCeMZLo9t9m9DFWGXyWhOWeRq8s/zUKJ1Hbt0iTiNzCFlbv7xM90UtXQ2 +#> +DCtgWYVAanZXdDeldy7bjvzFYdQzN9QxHkqnLGq+hFC +#> -----END CERTIFICATE----- +#> ',''),rs=c(10407,1189201696,17372321,1592963566,223318423,1152598252,1589836029))" +``` + +The printed value may be deployed directly on a remote machine. -The return value may be deployed manually on a remote machine by unescaping the double quotes around the call to `"mirai::daemon()"`, or directly via SSH or a resource manager by additionally specifying 'command' and 'args' to `launch_remote()`. +[« Back to ToC](#table-of-contents) #### CA Signed Certificates -As an alternative to the zero-configuration option, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to your organisation. +As an alternative to the zero-configuration default, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to an organisation. 1. Generate a private key and CSR. The following resources describe how to do so: - using Mbed TLS: - using OpenSSL: (Chapter 1.2 Key and Certificate Management) -2. Send or provide the generated CSR to the CA for it to sign a new TLS certificate. +2. Provide the generated CSR to the CA for it to sign a new TLS certificate. -- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, your TLS library should usually provide conversion utilities. -- Check also that your private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. +- The common name (CN) of the certificate must be identical to the hostname or IP address actually used for the connection as it is verified, and will fail if not the same. +- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, the TLS library used should usually provide conversion utilities. +- Check also that the private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. 3. When setting daemons, the TLS certificate and private key should be provided to the 'tls' argument of `daemons()`. - If the certificate and private key have been imported as character strings `cert` and `key` respectively, then the 'tls' argument may be specified as the character vector `c(cert, key)`. -- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this new file may be provided to the 'tls' argument. +- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this file may be provided to the 'tls' argument. 4. When launching daemons, the certificate chain to the CA should be supplied to the 'tls' argument of `daemon()` or `launch_remote()`. - The certificate chain should comprise multiple certificates, each between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers. The first one should be the newly-generated TLS certificate, the same supplied to `daemons()`, and the final one should be a CA root certificate. -- These are the only certificates required if your certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with your TLS certificate and ends with the certificate of the CA. -- If these are concatenated together as a single character string `certchain` (and assuming no certificate revocation list), then the character vector `c(certchain, "")` may be supplied to the relevant 'tls' argument. +- These are the only certificates required if the certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with the TLS certificate and ends with the certificate of the CA. +- If these are concatenated together as a single character string `certchain`, then the character vector comprising this and an empty character string `c(certchain, "")` may be supplied to the relevant 'tls' argument. - Alternatively, if these are written to a file (and the file replicated on the remote machines), then the 'tls' argument may also be specified as a path/filename (assuming these are the same on each machine). [« Back to ToC](#table-of-contents) @@ -567,3 +612,56 @@ is_error_value(m3$data) `is_error_value()` tests for all mirai execution errors, user interrupts and timeouts. [« Back to ToC](#table-of-contents) + +### Parallel Clusters + +This package contains experimental functions and S3 methods using `mirai` as an alternative communications backend for R. These were developed to fulfil a request by R Core at R Project Sprint 2023, and required R >= 4.4 (currently R-devel). + +`make_cluster()` creates a cluster of type 'miraiCluster', which may be used as a cluster object for any function in the `parallel` base package such as `parallel::clusterApply()` or `parallel::parLapply()`. + + + Function arguments 'n', 'url' and 'remote' map to `daemons()`, and remote launch configurations generated by `remote_config()` or `ssh_config()` may be used. + + Parallel clusters use synchronous load-balancing, hence dispatcher is not offered as an option. + + `status()` may be called on a 'miraiCluster` to return its connection status. + + +```r +cl <- make_cluster(4) +cl +#> < miraiCluster > +#> - cluster ID: `1` +#> - nodes: 4 +#> - active: TRUE + +parallel::parLapply(cl, 1:5, rnorm) +#> [[1]] +#> [1] -0.4992204 +#> +#> [[2]] +#> [1] -0.8198414 -0.5253678 +#> +#> [[3]] +#> [1] -0.12048654 0.07750113 0.21120964 +#> +#> [[4]] +#> [1] -0.1779141 0.6442720 1.1637351 0.7099819 +#> +#> [[5]] +#> [1] 0.4655363 0.4363602 0.2209481 1.1357764 0.8509574 + +status(cl) +#> $connections +#> [1] 1 +#> +#> $daemons +#> [1] "abstract://ce6bec24a44c1aea0ebd9799" + +stop_cluster(cl) +``` + +Created clusters are fully compatible with parallel cluster types. + +As an example, they may be registered by package [`doParallel`](https://cran.r-project.org/package=doParallel) for use with the [`foreach`](https://cran.r-project.org/package=foreach) package. + + +[« Back to ToC](#table-of-contents) + diff --git a/vignettes/reference.Rmd.orig b/vignettes/reference.Rmd.orig index 8e62fc418..96b4c121b 100644 --- a/vignettes/reference.Rmd.orig +++ b/vignettes/reference.Rmd.orig @@ -24,9 +24,11 @@ knitr::opts_chunk$set( 3. [Example 3: Resilient Pipelines](#example-3-resilient-pipelines) 4. [Daemons: Local Persistent Processes](#daemons-local-persistent-processes) 5. [Distributed Computing: Remote Daemons](#distributed-computing-remote-daemons) -6. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) -7. [Compute Profiles](#compute-profiles) -8. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) +6. [Distributed Computing: Launching Daemons](#distributed-computing-launching-daemons) +7. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections) +8. [Compute Profiles](#compute-profiles) +9. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts) +10. [Parallel Clusters](#parallel-clusters) ### Example 1: Compute-intensive Operations @@ -223,7 +225,7 @@ Requesting the status now shows 6 connections, along with the host URL at `$daem status() ``` -This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons remain idle. +This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons are idle having already completed their tasks. The advantage of this approach is that it is low-level and does not require an additional dispatcher process. It is well-suited to working with similar-length tasks, or where the number of concurrent tasks typically does not exceed available daemons. @@ -239,62 +241,31 @@ Set the number of daemons to zero to reset. The daemons interface may also be used to send tasks for computation to remote daemon processes on the network. -Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections. - -The examples below use an illustrative local network IP address of '10.75.32.74'. - -A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below. +Call `daemons()` specifying 'url' as a character string such as: 'tcp://10.90.127.129:5555' at which daemon processes should connect to. IPv6 addresses are also supported and must be enclosed in square brackets `[]` to avoid confusion with the final colon separating the port. For example, port 5555 on the IPv6 address `::ffff:a6f:50d` would be specified as `tcp://[::ffff:a6f:50d]:5555`. +For options on actually launching the daemons, please see the next section. + #### Connecting to Remote Daemons Through Dispatcher The default `dispatcher = TRUE` creates a background `dispatcher()` process on the local machine, which listens to a vector of URLs that remote `daemon()` processes dial in to, with each daemon having its own unique URL. It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port. -```{r localqueue} -daemons(n = 4, url = "ws://10.75.32.74:5555") -``` - -Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs. - -Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.: +Supplying a vector of URLs allows the use of arbitrary port numbers / paths. 'n' does not need to be specified if it can be inferred from the length of the 'url' vector, for example: ```{r vectorqueue, eval=FALSE} -daemons(url = c("ws://10.75.32.74:5566/cpu", "ws://10.75.32.74:5566/gpu", "ws://10.75.32.74:7788/1")) -``` - -Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied. - --- - -On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at: - -``` -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/1")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/2")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/3")' -Rscript -e 'mirai::daemon("ws://10.75.32.74:5555/4")' - +daemons(url = c("ws://10.90.127.129:5566/cpu", "ws://10.90.127.129:5566/gpu", "ws://10.90.127.129:7788/1")) ``` -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. +Alternatively, below a single URL is supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce the URLs. -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: - -```{r launchremote, eval=FALSE} -launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.32.100", .)) -``` -```{r launchremotereal, echo=FALSE} -launch_remote(1:4) +```{r localqueue} +daemons(n = 4, url = "ws://10.90.127.129:5555") ``` -The returned vector comprises the shell commands executed on the remote machine. - --- - -Requesting status, on the host machine: +Requesting status on the host machine: ```{r remotev2} status() @@ -322,55 +293,91 @@ Closing the connection causes the dispatcher to exit automatically, and in turn By specifying `dispatcher = FALSE`, remote daemons connect directly to the host process. The host listens at a single URL, and distributes tasks to all connected daemons. -```{r remote, eval=FALSE} -daemons(url = "tcp://10.75.32.74:0", dispatcher = FALSE) -``` - -Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example: - -```{r remotealt} -daemons(url = "tcp://:0", dispatcher = FALSE) +```{r remote} +daemons(url = "tcp://10.90.127.129:0", dispatcher = FALSE) ``` Note that above, the port number is specified as zero. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`. --- +The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. -On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks: +`$connections` will show the actual number of connected daemons. +```{r remotev} +status() ``` -Rscript -e 'mirai::daemon("tcp://10.75.32.74:0")' + +To reset all connections and revert to default behaviour: + +```{r reset} +daemons(0) ``` -Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available. -`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.32.100 accepts SSH connections over port 22: +This causes all connected daemons to exit automatically. + +[« Back to ToC](#table-of-contents) + +### Distributed Computing: Launching Daemons + +To launch remote daemons, supply a remote launch configuration to the 'remote' argument of `daemons()` when setting up daemons, or `launch_remote()` at any time afterwards. -```{r launchremoteu, eval=FALSE} -launch_remote("tcp://10.75.32.74:0", command = "ssh", args = c("-p 22 10.75.32.100", .)) +`ssh_config()` may be used to generate a remote launch configuration if there is SSH access to the remote machine, or else `remote_config()` provides a flexible method for generating a configuration involving a custom resource manager / application. + +#### SSH Direct Connection + +The first example below launches 4 daemons on the machine 10.75.32.90 (using the default SSH port of 22 as this was not specified), connecting back to the dispatcher URLs: + +```{r ldmn, eval=FALSE} +daemons( + n = 4, + url = "ws://10.75.32.74:5555", + remote = ssh_config(remotes = "ssh://10.75.32.90") +) ``` -```{r launchremoteureal, echo=FALSE} -launch_remote("tcp://10.75.32.74:0") +The second example below launches one daemon on each of 10.75.32.90 and 10.75.32.91 using the custom SSH port of 222: + +```{r ldmnd, eval=FALSE} +daemons( + n = 2, + url = "ws://10.75.32.74:5555", + remote = ssh_config(c("ssh://10.75.32.90:222", "ssh://10.75.32.91:222")) +) ``` +In the above examples, as the remote daemons connect back directly, port 5555 on the local machine must be open to incoming connections from the remote addresses. -The returned vector comprises the shell commands executed on the remote machine. +#### SSH Tunnelling --- +Use of SSH tunnelling provides a convenient way to launch remote daemons without requiring the remote machine to be able to access the host. Often firewall configurations or security policies may prevent opening a port to accept outside connections. -The number of daemons connecting to the host URL is not limited and network resources may be added or removed at any time, with tasks automatically distributed to all connected daemons. +In these cases SSH tunnelling offers a solution by creating a tunnel once the initial SSH connection is made. For simplicity, this SSH tunnelling implementation uses the same port on both the side of the host and that of the corresponding node. SSH key-based authentication must also already be in place. -`$connections` will show the actual number of connected daemons. +Tunnelling requires the hostname for 'url' specified when setting up daemons to be either 'localhost' or '127.0.0.1'. This is as the tunnel is created between localhost:port or equivalently 127.0.0.1:port on each machine. The host listens to its localhost:port and the remotes each dial into localhost:port on their own respective machines. -```{r remotev} -status() +The below example launches 2 nodes on the remote machine 10.75.32.90 using SSH tunnelling over port 5555 ('url' hostname is specified as 'localhost'): + +```{r sshrevtun, eval=FALSE} +daemons( + url = "tcp://localhost:5555", + remote = ssh_config( + remotes = c("ssh://10.75.32.90", "ssh://10.75.32.90"), + tunnel = TRUE + ) +) ``` -To reset all connections and revert to default behaviour: +#### Manual Deployment + +As an alternative to automated launches, calling `launch_remote()` without specifying 'remote' may be used to return the shell commands for deploying daemons manually. The printed return values may be copy / pasted directly to a remote machine. + +```{r launchremotereal} +daemons(n = 2, url = "tcp://[::1]:0") + +launch_remote(1:2) -```{r reset} daemons(0) ``` -This causes all connected daemons to exit automatically. +Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying the argument `asyncdial = TRUE` will allow daemons to wait (indefinitely) for a connection to become available. [« Back to ToC](#table-of-contents) @@ -394,36 +401,39 @@ The generated self-signed certificate is available via `launch_remote()`. This f launch_remote(1) ``` -The return value may be deployed manually on a remote machine by unescaping the double quotes around the call to `"mirai::daemon()"`, or directly via SSH or a resource manager by additionally specifying 'command' and 'args' to `launch_remote()`. +The printed value may be deployed directly on a remote machine. ```{r tlsclose, include=FALSE} daemons(0) ``` +[« Back to ToC](#table-of-contents) + #### CA Signed Certificates -As an alternative to the zero-configuration option, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to your organisation. +As an alternative to the zero-configuration default, a certificate may also be generated via a Certificate Signing Request (CSR) to a Certificate Authority (CA), which may be a public CA or a CA internal to an organisation. 1. Generate a private key and CSR. The following resources describe how to do so: - using Mbed TLS: - using OpenSSL: (Chapter 1.2 Key and Certificate Management) -2. Send or provide the generated CSR to the CA for it to sign a new TLS certificate. +2. Provide the generated CSR to the CA for it to sign a new TLS certificate. -- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, your TLS library should usually provide conversion utilities. -- Check also that your private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. +- The common name (CN) of the certificate must be identical to the hostname or IP address actually used for the connection as it is verified, and will fail if not the same. +- The received certificate should comprise a block of cipher text between the markers `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----`. Make sure to request the certificate in the PEM format. If only available in other formats, the TLS library used should usually provide conversion utilities. +- Check also that the private key is a block of cipher text between the markers `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----`. 3. When setting daemons, the TLS certificate and private key should be provided to the 'tls' argument of `daemons()`. - If the certificate and private key have been imported as character strings `cert` and `key` respectively, then the 'tls' argument may be specified as the character vector `c(cert, key)`. -- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this new file may be provided to the 'tls' argument. +- Alternatively, the certificate may be copied to a new text file, with the private key appended, in which case the path/filename of this file may be provided to the 'tls' argument. 4. When launching daemons, the certificate chain to the CA should be supplied to the 'tls' argument of `daemon()` or `launch_remote()`. - The certificate chain should comprise multiple certificates, each between `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers. The first one should be the newly-generated TLS certificate, the same supplied to `daemons()`, and the final one should be a CA root certificate. -- These are the only certificates required if your certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with your TLS certificate and ends with the certificate of the CA. -- If these are concatenated together as a single character string `certchain` (and assuming no certificate revocation list), then the character vector `c(certchain, "")` may be supplied to the relevant 'tls' argument. +- These are the only certificates required if the certificate was signed directly by a CA. If not, then the intermediate certificates should be included in a certificate chain that starts with the TLS certificate and ends with the certificate of the CA. +- If these are concatenated together as a single character string `certchain`, then the character vector comprising this and an empty character string `c(certchain, "")` may be supplied to the relevant 'tls' argument. - Alternatively, if these are written to a file (and the file replicated on the remote machines), then the 'tls' argument may also be specified as a path/filename (assuming these are the same on each machine). [« Back to ToC](#table-of-contents) @@ -478,3 +488,32 @@ is_error_value(m3$data) `is_error_value()` tests for all mirai execution errors, user interrupts and timeouts. [« Back to ToC](#table-of-contents) + +### Parallel Clusters + +This package contains experimental functions and S3 methods using `mirai` as an alternative communications backend for R. These were developed to fulfil a request by R Core at R Project Sprint 2023, and required R >= 4.4 (currently R-devel). + +`make_cluster()` creates a cluster of type 'miraiCluster', which may be used as a cluster object for any function in the `parallel` base package such as `parallel::clusterApply()` or `parallel::parLapply()`. + + + Function arguments 'n', 'url' and 'remote' map to `daemons()`, and remote launch configurations generated by `remote_config()` or `ssh_config()` may be used. + + Parallel clusters use synchronous load-balancing, hence dispatcher is not offered as an option. + + `status()` may be called on a 'miraiCluster` to return its connection status. + +```{r parcluster} +cl <- make_cluster(4) +cl + +parallel::parLapply(cl, 1:5, rnorm) + +status(cl) + +stop_cluster(cl) +``` + +Created clusters are fully compatible with parallel cluster types. + +As an example, they may be registered by package [`doParallel`](https://cran.r-project.org/package=doParallel) for use with the [`foreach`](https://cran.r-project.org/package=foreach) package. + + +[« Back to ToC](#table-of-contents) +