running workflows may appear stopped on network filesystems

[oliver-sanders]

Regression of: #2943
Bug report: https://cylc.discourse.group/t/cylc-set-slow-to-be-able-to-run-after-workflow-started/1073/5

On network filesystems, there is a lag between a file being created on one host and it appearing on another.

The Cylc contact file contains the details of the running scheduler and is how we detect if a workflow is running or not.

Through a combination of running fsync on the contact file, and listing the directory it is contained in, we can force the network filesystem to synchronize the file reducing this lag.

By default, Cylc installs each run of a workflow into a numbered directory (run1, run2, ...) with a runN` symlink pointing at the most recent run number. It would appear that we need to perform a filesystem listing on this symlink before the contact file is synchronized to other nodes on the network.

There is another symlink that might trip things up which is configured by global.cylc[install][symlink dirs]run. We will likely need to perform an additional filesystem listing if this symlink is created.

[hjoliver]

(Calling this a bug in Cylc seems a little harsh! but fair enough)