| title |
|---|
Preparations for participating in the tutorial |
This page includes some information on how to prepare for participating in this tutorial.
Minimal requirements:
- Basic understanding of your preferred command line interface (ie a bash terminal)
- Access to CMIP data
Optional, but useful:
- Basic understanding of git
- Access to a suitable computing system (eg CEDA-Jasmin, DKRZ-Mistral)
- GitHub account {: .prereq}
We typically use the command line to interact with ESMValTool. While most of us are likely to have experience with the command line, novices may want to work through this software carpentry unix shell course.
- Command line: https://swcarpentry.github.io/shell-novice/
Git is a distributed version-control system for tracking changes in source code during software development. It’s how we distribute, share, and manage the ESMValTool code.
To complete this tutorial and use ESMValTool, you will need access to data in a reasonable format. Some data will be provided, but there is simply too much data available for your tutors to make it all available directly.
ESMValTool may be run on multiple platforms, from your local machine to large computing clusters. The best option is to use a computing cluster with an Earth System Grid Federation (ESGF) node. The benefit of using a compute cluster with an ESGF node is that the Coupled Model Intercomparison Project (CMIP) is locally stored on disk and accessible directly by the tool. Similarly, observational data would also be available at these sites. The ESGF also hosts observations for Model Intercomparison Projects (obs4MIPs) and reanalyses data (ana4MIPs).
Here are a few options for compute clusters with ESGF nodes:
For more information see:
-
CMIP5 and CMIP6 data obey the CF-conventions. Available variables can be found under the CMIP5 data request and the CMIP6 Data Request.
-
List of all CMIP named variables.
-
List of all ESGF nodes.
-
A good tutorial on how to search and download CMIP data from ESGF nodes.
-
Exploring climate model data on infrastructure for the European network for Earth system modelling.
Please skip this section if you are not going to use JASMIN and go here.
If you do not already have an account on JASMIN, then request an account as soon as possible. Please follow these instructions on how to create a Jasmin account
During the account creation, you will need an SSH key, which can be generated following these instructions
Here are some more instructions on how to get started with jasmin.
Also note that if you are working from home, JASMIN may not be directly accessible from your home. You may need to use ssh to connect to a machine in your institute and then on to JASMIN. Please test your connection before the tutorial.
Note that you have only created an account for the web-interface. To log into the jasmin machine and do work, you'll need to create a login account too using this page.
Please request access to the working groups:
Once you have access to the data archive on CEDA, make sure to link your CEDA and JASMIN accounts. This can be done by checking the link to CEDA box on your JASMIN profile page.
The linking may take a few hours to take effect and is necessary for you to access the BADC archives via JASMIN. Some CMIP5 data sets such as MIROC are not accessible by default and special permission has to be requested to access them via the CEDA catalogue page.
Log into jasmin-login:
ssh -X [email protected]
{: .language-bash}
Then log into the sci1 machine:
ssh -X jasmin-sci1
{: .language-bash}
Can you see the following locations:
ls /gws/nopw/j04/esmeval/obsdata-v2/
ls /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES
ls /badc/cmip6/data/CMIP6/CMIP/*/*/historical/r1i1p1f?/Omon/[ts]os/gn/latest/*.nc
{: .language-bash}
Note that the JASMIN is only open to certain locations (mostly universities, and research centres). You may need a VPN if you wish to connect from your home network.
Please request access to the working groups:
Once you have access to the data archive on CEDA, make sure to link your CEDA and JASMIN accounts. This can be done by checking the link to CEDA box on your JASMIN profile page.
The linking may take a few hours to take effect and is necessary for you to access the BADC archives via JASMIN. Some CMIP5 data sets such as MIROC are not accessible by default and special permission has to be requested to access them via the CEDA catalogue page.
Congratulations! Please go here here next.
Please skip this section if you are not going to use DKRZ and go here.
If you do not already have an account at the DKRZ, then register as soon as possible. You could find a short introduction on how to get started at DKRZ here.
There is also a user manual for Levante which is DKRZ's current supercomputer.
To use the resources on DKRZ you have to join a project. One option is to join an existing project by logging into https://luv.dkrz.de/ with your account and select 'Join existing project'. Once you are accepted by the manager of your chosen project, your web account will be turned into a full LDAP account which will allow you to log into and use the DKRZ's resources. If you do not have access to an existing project, another option for you would be to apply for resources at DKRZ. Here are some instructions on how to apply for resources.
CMIP5 and CMIP6 data are available in these directories:
- CMIP5: /work/kd0956/CMIP5/data/cmip5/output1/
- CMIP6: /work/ik1017/CMIP6/data/CMIP6/CMIP/
Log into Levante (DKRZ)
ssh -X [email protected]
{: .language-bash}
Login nodes are for compiling and job submission only! For all other tasks, you can use the [interactive](https://docs.dkrz.de/doc/levante/running-jobs/ partitions-and-limits.html#interactive) partition or start an interactive session.
Data storage:
- Personal data: home directory (30GiB)
- Project data: /work/project_id/username
- Temporary data: scratch directory on /scratch/*/username is automatically deleted after 14 days (15TiB) (Please use this directory for all your testing! Do not use the work directory for tests.) (see also this)
Running batch jobs: Info and examples on SLURM job scheduling system at DKRZ can be found here.
Congratulations! Please go here here next.
Please skip this section if you are not going to use ESMValTool on your local machine and go here.
If you are planning on running ESMValTool on your own machine, please make sure that you are able to download CMIP data and that you have a few GB of space available to install conda and ESMValTool, but also enough to make a copy of some data (~125MB) needed for this tutorial.
You can use ESMValTool to automatically download data needed for test recipes. Please see the [Configuration][lesson-configuration] episode or the [configuration file documentation][config-file] for more information. This the recommended option as it has the advantage that data is stored in subdirectories, and features such as wildcards and recording the version of the data will work automatically.
Alternatively, you can run the following command using wget:
wget --no-clobber --input-file \
https://github.com/ESMValGroup/ESMValTool_Tutorial/raw/main/data/dataset.urls \
--directory-prefix $HOME/esmvaltool_tutorial/data/You don’t need a github account to participate in the tutorial. However, if you want to raise an issue, contribute to the discussions, or share your code, please create a github account.
To learn how to use github, please have a look at this introduction to github.
You may hear a few of the following phrases during the tutorial. Don’t be alarmed, they will make sense eventually.
Issues are github’s ticketing system. They allow users and developers to discuss problems, identify bugs, or to make suggestions. Each issue is assigned a number and will have it’s own page on GitHub.
Here’s an explanation of the GitHub issues.
Raising an issue is the act of creating a new issue. If you are asked to raise an issue, please follow any instructions that you are given, and also make sure that you read the default issue text.
A GitHub pull request is the act of requesting that a branch is merged with another branch.
This is an advanced feature of GitHub, and will generally be performed by the ESMValTool development team.
{% include links.md %}