JASMIN

The JASMIN platform is the Natural Environment Research Council (NERC)'s compute service. If you are interested in its metrics, look here.

The start of JASMIN's documentation is here. There are video instructions.

There are a set of exercises for getting started from a workshop.

Note: in the process of being updated: This GitHub has recordings for various workshops with accompanying exercises, in particular:

• Accessing and using the JASMIN “scientific analyis” severs
• Copy data to JASMIN to share with collaborators
• Processing on the scientific analysis servers
• A script to extract 1 variable from a file in the CEDA Archive
• Batch run a script on LOTUS
• Choosing the right storage for your workflow
• Manage a multi-step workflow
• Building your own Python 3 environment

Warwick HPC for Data Science video lecture series

1. Signing up for JASMIN access - Account and services

To access JASMIN, you must sign up for an account following the instructions here: https://help.jasmin.ac.uk/article/4435-get-a-jasmin-account

In the 'What will you be using JASMIN for' box, put something along the lines of:

Environmental data analysis tasks forming part of my MRes training in the Artificial Intelligence for Environmental Risk (AI4ER, https://ai4er-cdt.esc.cam.ac.uk/) Centre for Doctoral Training hosted at the University of Cambridge and the British Antarctic Survey.
My supervisor is SUPERVISOR_NAME ([email protected]).

Once you've confirmed your email address, you'll be asked for some account details, including an SSH Public Key. If you do not yet have an SSH Public key follow this guide to set up an SSH key pair. If using MobaXterm on Windows, first choose a persistent home directory in Settings/ Configuration / General for MobaXterm as explained here.

Your account so far is just a profile on the accounts portal but has no privileges. If you intend to use JASMIN computing resources you need to apply for the jasmin-login service, which will allow you to connect to JASMIN machines using ssh. When asked for supporting information provide the same information as when applying for the account. This service allocates you a home area of 100GB, and you can access the login servers using SSH. You can submit jobs to LOTUS using the batch scheduler SLURM. To submit python scripts, this example of submitting a script might be useful, and the JASMIN documentation provides a beginner example

You can apply to access our 10TB Group Work Space (GWS) following these instructions. The ai4er group work space can be found by searching for ai4er or following this link. For supporting information provide the same information as when applying for the account.

It is no longer necessary to apply for the Jupyter Notebook service. It is available to all users holding the jasmin-login user role. Follow instructions here.

The approval process can take a couple of days.

2. Signing up for CEDA access

To access the CEDA data archive you need a CEDA account. You register here. After this you need to link your CEDA account to your JASMIN account. To do so, when logged into the JASMIN account portal and on your profile page, click the "link now" button. Once linked you are a member of the Unix group open. To check just type groups. You can apply for access to further data sets. This post describes how to use the CEDA data catalogue.

3. Logging in to JASMIN servers

JASMIN restricts login access by maintaining an "allow list" of network domains which are allowed to make SSH connections to the JASMIN login gateways. All .ac.uk network domains are already registered. Thus access JASMIN from an .ac.uk network: https://help.jasmin.ac.uk/article/190-check-network-details. If connecting from home, connect to your institution via VPN before making an outgoing SSH connection to a JASMIN login server. The wiki page on the Cambridge VPN will help you in setting up the VPN service on your device if you have not set it up yet.

Log-in instructions are here: https://help.jasmin.ac.uk/article/187-login. If encountering "Could not add identity "~.ssh/id_rsa_jasmin": agent refused operation", it might already be loaded. Check with ssh-add -l. Otherwise, check whether ssh-agent is running with Get-Service ssh-agent or if on Windows type services in the command box and look for OpenSSH Authentication Agent. If this still does not resolve the problem check whether the permissions on the key file are not too open. Further trouble shooting https://help.jasmin.ac.uk/article/848-login-problems.

Note that you cannot do any actual work on the login servers. You need to log in to one of the scientific analysis servers https://help.jasmin.ac.uk/article/121-sci-servers.

4. Group Work Spaces (GWS)

The introduction is found here https://help.jasmin.ac.uk/article/199-introduction-to-group-workspaces. https://help.jasmin.ac.uk/article/3839-gws-etiquette describes how information about data in a GWS should be shared. Please follow the instruction on directory structure and meta data.

The path for the AI4ER GWS is /gws/nopw/j04/ai4er. Members of the UNIX group gws_ai4er have access. Instructions on creating directories and making contents writable to other users are here https://help.jasmin.ac.uk/article/203-managing-a-gws. Note that the default set of permissions is set to umask 002 meaning that members of our group can create and modify data files.

TO DO: Showcases can be made accessible to a wider audience via https://help.jasmin.ac.uk/article/202-share-gws-data-via-http. Back-up arrangements.

4.1 Data transfer

A list of data transfer help pages is here: https://help.jasmin.ac.uk/category/217-data-transfer. A list of transfer servers is here: https://help.jasmin.ac.uk/article/192-transfer-servers. rsync (https://help.jasmin.ac.uk/article/3810-data-transfer-tools-rsync-scp-sftp) is recommended, because it is simple and generally applicable. Remember -v for verbose, -r recursive for directories.

5. Python 3

JASMIN has some built in virtual environments under module load jaspy: https://help.jasmin.ac.uk/article/4729-jaspy-envs.

If you want, you can use venv to install your packages on top of this as detailed here. However, many things are simplified, particularly using the batch computing cluster, if you simply install packages to your home directory. You can install conda there and setup conda so that it automatically activates when you SSH in, or when you run a job on the LOTUS cluster. Instructions to do this are in the Conda configuration tutorial in the sidebar.

If you are using Matplotlib to visualise data, there is some advice here: https://help.jasmin.ac.uk/article/4733-matplotlib. Especially on the batch server LOTUS, you need to set the rendering backend matplotlib.use('agg') before importing matplotlib.pyplot (https://matplotlib.org/faq/usage_faq.html#what-is-a-backend).

6. Jupyter Notebook Service

JASMIN has a special Jupyter notebook service, as detailed here: https://help.jasmin.ac.uk/article/4851-jasmin-notebook-service. However, it is often easier to just install Jupyter in a virtual environment in your home directory, and run it from there. You can then use port forwarding to access the notebook from your computer, or use VS Code to develop on the JASMIN Sci servers with its built-in notebook editor.

If you wish to use JASMIN's notebook service, an introduction is here https://github.com/cedadev/ceda-notebooks/blob/master/notebooks/intro/notebook-tour.ipynb. Once you have the JupyterLab interface, the left hand panel will show your home directory. For security reasons, it is not possible for a user to open a notebook from outside of their home directory. Notebooks can access files from the CEDA Archive, and also have read-only access to group workspaces. Launch a new notebook and write the code

with open('/gws/nopw/j04/ai4er/wiki_link.txt') as file: # file closed after with block
    print(file.read())

to check that you can access the GWS from the notebook service.

You can execute shell commands in a Python console, or in a Python cell in a notebook by prefixing the line with !, e.g. ! ls /gws/nopw/j04/ai4er. You can copy notebooks into your home directory to work on and run them, e.g. ! cp /gws/nopw/j04/ai4er/notebook-examples/plot_polar_era5_hourly_mslp_t2m_and_uv10.ipynb $HOME.

plot_polar_era5_hourly_mslp_t2m_and_uv10.ipynb (kindly provided by Tony Phillips) makes a polar stereographic plot of ERA5 hourly MSLP, 2m temperature and/or 10m wind Further example notebooks are at https://github.com/cedadev/ceda-notebooks/blob/master/notebooks. Remember that you need a linked CEDA account.

7. Available Servers:

The following servers are available via jasmin, given you have the right permissions:

• login = login servers: jasmin-login1.ceda.ac.uk, cems-login1.cems.ac.uk
• sci = scientific analysis servers: jasmin-sci[1-5].ceda.ac.uk, cems-sci[1-2].cems.rl.ac.uk
• transfer = data transfer servers: jasmin-xfer[1-2].ceda.ac.uk, cems-xfer1.cems.rl.ac.uk
• LOTUS = LOTUS batch processing cluster (all cluster nodes)

For more info on the specification of the scientific analysis servers, see the following table (table taken from here)

Server name	Virtual/Physical (OperatingSystem)	Processor model	CPU Cores	RAM (GB)	/tmp (GB)
sci1.jasmin.ac.uk	V (CentOS7)	Xeon(R) Silver 4114 CPU @ 2.20GHz	8	16	NA
sci2.jasmin.ac.uk	V (CentOS7)	Xeon(R) E5-2630 @ 2.20GHz	8	16	NA
sci3.jasmin.ac.uk	P (CentOS7)	AMD EPYC 7402	48	1000	-
sci4.jasmin.ac.uk	V (CentOS7)	Xeon E5-2660 @ 2.20GHz	8	32	NA
sci5.jasmin.ac.uk	V (CentOS7)	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz	8	32	NA
sci6.jasmin.ac.uk	P (CentOS7)	AMD EPYC 7402	48	1000	-
sci8.jasmin.ac.uk	P (CentOS7)	Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz	24	383	-

8. Convenient SSH access:

This quick guide explains how to set up a convenient SSH access to Jasmin (via the SSH config file and proxy jumps) that makes developing on Jasmin feel like you're developing on your local machine.

9. JASMIN GPU Tutorial

This guide explains how to use the JASMIN GPU nodes. Note that you first need to request access to the GPU cluster from JASMIN support.