diff --git a/README.md b/README.md index 71d0f35..19cf812 100644 --- a/README.md +++ b/README.md @@ -2,27 +2,27 @@ # stackhpc.openhpc -This Ansible role installs packages and performs configuration to provide an OpenHPC v2.x Slurm cluster. +This Ansible role installs packages and performs configuration to provide a Slurm cluster. By default this uses packages from [OpenHPC](https://openhpc.community/) but it is also possible to use alternative Slurm binaries and packages. As a role it must be used from a playbook, for which a simple example is given below. This approach means it is totally modular with no assumptions about available networks or any cluster features except for some hostname conventions. Any desired cluster fileystem or other required functionality may be freely integrated using additional Ansible roles or other approaches. The minimal image for nodes is a RockyLinux 8 GenericCloud image. +## Task files +This role provides four task files which can be selected by using the `tasks_from` parameter of Ansible's `import_role` or `include_role` modules: +- `main.yml`: Runs `install-ohpc.yml` and `runtime.yml`. Default if no `tasks_from` parameter is used. +- `install-ohpc.yml`: Installs repos and packages for OpenHPC. +- `install-generic.yml`: Installs systemd units etc. for user-provided binaries. +- `runtime.yml`: Slurm/service configuration. + ## Role Variables +Variables only relevant for `install-ohpc.yml` or `install-generic.yml` task files are marked as such below. + `openhpc_extra_repos`: Optional list. Extra Yum repository definitions to configure, following the format of the Ansible -[yum_repository](https://docs.ansible.com/ansible/2.9/modules/yum_repository_module.html) module. Respected keys for -each list element: -* `name`: Required -* `description`: Optional -* `file`: Required -* `baseurl`: Optional -* `metalink`: Optional -* `mirrorlist`: Optional -* `gpgcheck`: Optional -* `gpgkey`: Optional - -`openhpc_slurm_service_enabled`: boolean, whether to enable the appropriate slurm service (slurmd/slurmctld). +[yum_repository](https://docs.ansible.com/ansible/2.9/modules/yum_repository_module.html) module. + +`openhpc_slurm_service_enabled`: Optional boolean, whether to enable the appropriate slurm service (slurmd/slurmctld). Default `true`. `openhpc_slurm_service_started`: Optional boolean. Whether to start slurm services. If set to false, all services will be stopped. Defaults to `openhpc_slurm_service_enabled`. @@ -30,7 +30,7 @@ each list element: `openhpc_slurm_control_host_address`: Optional string. IP address or name to use for the `openhpc_slurm_control_host`, e.g. to use a different interface than is resolved from `openhpc_slurm_control_host`. -`openhpc_packages`: additional OpenHPC packages to install. +`openhpc_packages`: Optional list. Additional OpenHPC packages to install (`install-ohpc.yml` only). `openhpc_enable`: * `control`: whether to enable control host @@ -46,7 +46,19 @@ each list element: `openhpc_login_only_nodes`: Optional. If using "configless" mode specify the name of an ansible group containing nodes which are login-only nodes (i.e. not also control nodes), if required. These nodes will run `slurmd` to contact the control node for config. -`openhpc_module_system_install`: Optional, default true. Whether or not to install an environment module system. If true, lmod will be installed. If false, You can either supply your own module system or go without one. +`openhpc_module_system_install`: Optional, default true. Whether or not to install an environment module system. If true, lmod will be installed. If false, You can either supply your own module system or go without one (`install-ohpc.yml` only). + +`openhpc_generic_packages`: Optional. List of system packages to install, see `defaults/main.yml` for details (`install-generic.yml` only). + +`openhpc_sbin_dir`: Optional. Path to slurm daemon binaries such as `slurmctld`, default `/usr/sbin` (`install-generic.yml` only). + +`openhpc_bin_dir`: Optional. Path to Slurm user binaries such as `sinfo`, default `/usr/bin` (`install-generic.yml` only). + +`openhpc_lib_dir`: Optional. Path to Slurm libraries, default `/usr/lib64/slurm` (`install-generic.yml` only). + +`openhpc_config_files`: Optional. List of additional Slurm configuration files to template. Changes to any templated files will restart `slurmctld` and `slurmd`s. The default templates `gres.conf` on the control node. List elements are dicts which must contain: + - `template`: A dict with parameters for Ansible's [template](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html) module. + - `enable`: String `control`, `batch`, `database` or `runtime` specifying nodes to template this file on (i.e. matches keys from `openhpc_enable`). Any other string results in no templating. ### slurm.conf @@ -79,12 +91,18 @@ For each group (if used) or partition any nodes in an ansible inventory group `< `openhpc_cluster_name`: name of the cluster. -`openhpc_config`: Optional. Mapping of additional parameters and values for `slurm.conf`. Note these will override any included in `templates/slurm.conf.j2`. +`openhpc_config`: Optional. Mapping of additional parameters and values for `slurm.conf`. Note these will override any included in `templates/slurm.conf.j2`. Setting a parameter's value to the string `` will omit a parameter which is included in the template. `openhpc_ram_multiplier`: Optional, default `0.95`. Multiplier used in the calculation: `total_memory * openhpc_ram_multiplier` when setting `RealMemory` for the partition in slurm.conf. Can be overriden on a per partition basis using `openhpc_slurm_partitions.ram_multiplier`. Has no effect if `openhpc_slurm_partitions.ram_mb` is set. `openhpc_state_save_location`: Optional. Absolute path for Slurm controller state (`slurm.conf` parameter [StateSaveLocation](https://slurm.schedmd.com/slurm.conf.html#OPT_StateSaveLocation)) +`openhpc_slurmd_spool_dir`: Optional. Absolute path for slurmd state (`slurm.conf` parameter [SlurmdSpoolDir](https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdSpoolDir)) + +`openhpc_slurm_conf_template`: Optional. Path of Jinja template for slurm.conf configuration file. Default is `slurm.conf.j2` template in role. **NB:** The required templating is complex, if just setting specific parameters use `openhpc_config` intead. + +`openhpc_slurm_conf_path`: Optional. Path to template `slurm.conf` configuration file to. Default `/etc/slurm/slurm.conf` + #### Accounting By default, no accounting storage is configured. OpenHPC v1.x and un-updated OpenHPC v2.0 clusters support file-based accounting storage which can be selected by setting the role variable `openhpc_slurm_accounting_storage_type` to `accounting_storage/filetxt`[1](#slurm_ver_footnote). Accounting for OpenHPC v2.1 and updated OpenHPC v2.0 clusters requires the Slurm database daemon, `slurmdbd` (although job completion may be a limited alternative, see [below](#Job-accounting). To enable accounting: diff --git a/defaults/main.yml b/defaults/main.yml index c806809..c3bc31e 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -16,6 +16,17 @@ openhpc_gres_template: gres.conf.j2 openhpc_slurm_configless: "{{ 'enable_configless' in openhpc_config.get('SlurmctldParameters', []) }}" openhpc_state_save_location: /var/spool/slurm +openhpc_slurmd_spool_dir: /var/spool/slurm +openhpc_slurm_conf_path: /etc/slurm/slurm.conf +openhpc_slurm_conf_template: slurm.conf.j2 +openhpc_config_files: + - template: + dest: "{{ openhpc_slurm_conf_path | dirname }}/gres.conf" + src: "{{ openhpc_gres_template }}" + mode: "0600" + owner: slurm + group: slurm + enable: control # Accounting openhpc_slurm_accounting_storage_host: "{{ openhpc_slurmdbd_host }}" @@ -45,6 +56,15 @@ openhpc_enable: database: false runtime: false +# Only used for install-generic.yml: +openhpc_generic_packages: + - munge + - mariadb-connector-c # only required on slurmdbd + - hwloc-libs # only required on slurmd +openhpc_sbin_dir: /usr/sbin # path to slurm daemon binaries (e.g. slurmctld) +openhpc_bin_dir: /usr/bin # path to slurm user binaries (e.g sinfo) +openhpc_lib_dir: /usr/lib64/slurm # path to slurm libraries + # Repository configuration openhpc_extra_repos: [] @@ -92,12 +112,12 @@ ohpc_default_extra_repos: gpgcheck: true gpgkey: "https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8" -# Concatenate all repo definitions here -ohpc_repos: "{{ ohpc_openhpc_repos[ansible_distribution_major_version] + ohpc_default_extra_repos[ansible_distribution_major_version] + openhpc_extra_repos }}" +# Concatenate extra repo definitions here +ohpc_extra_repos: "{{ ohpc_default_extra_repos[ansible_distribution_major_version] + openhpc_extra_repos }}" openhpc_munge_key: openhpc_login_only_nodes: '' -openhpc_module_system_install: true +openhpc_module_system_install: true # only works for install-ohpc.yml/main.yml # Auto detection openhpc_ram_multiplier: 0.95 diff --git a/tasks/install-generic.yml b/tasks/install-generic.yml new file mode 100644 index 0000000..a767797 --- /dev/null +++ b/tasks/install-generic.yml @@ -0,0 +1,72 @@ +- include_tasks: pre.yml + +- name: Create a list of slurm daemons + set_fact: + _ohpc_daemons: "{{ _ohpc_daemon_map | dict2items | selectattr('value') | items2dict | list }}" + vars: + _ohpc_daemon_map: + slurmctld: "{{ openhpc_enable.control }}" + slurmd: "{{ openhpc_enable.batch }}" + slurmdbd: "{{ openhpc_enable.database }}" + +- name: Ensure extra repos + ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module] + loop: "{{ openhpc_extra_repos }}" + loop_control: + label: "{{ item.name }}" + +- name: Install system packages + dnf: + name: "{{ openhpc_generic_packages }}" + +- name: Create Slurm user + user: + name: slurm + comment: SLURM resource manager + home: /etc/slurm + shell: /sbin/nologin + +- name: Create Slurm unit files + template: + src: "{{ item }}.service.j2" + dest: /lib/systemd/system/{{ item }}.service + owner: root + group: root + mode: ug=rw,o=r + loop: "{{ _ohpc_daemons }}" + register: _slurm_systemd_units + +- name: Get current library locations + shell: + cmd: "ldconfig -v | grep -v ^$'\t'" # noqa: no-tabs risky-shell-pipe + register: _slurm_ldconfig + changed_when: false + +- name: Add library locations to ldd search path + copy: + dest: /etc/ld.so.conf.d/slurm.conf + content: "{{ openhpc_lib_dir }}" + owner: root + group: root + mode: ug=rw,o=r + when: openhpc_lib_dir not in _ldd_paths + vars: + _ldd_paths: "{{ _slurm_ldconfig.stdout_lines | map('split', ':') | map('first') }}" + +- name: Reload Slurm unit files + # Can't do just this from systemd module + command: systemctl daemon-reload # noqa: command-instead-of-module no-changed-when no-handler + when: _slurm_systemd_units.changed + +- name: Prepend $PATH with slurm user binary location + lineinfile: + path: /etc/environment + line: "{{ new_path }}" + regexp: "^{{ new_path | regex_escape }}" + owner: root + group: root + mode: u=gw,go=r + vars: + new_path: PATH="{{ openhpc_bin_dir }}:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin" + +- meta: reset_connection # to get new environment diff --git a/tasks/install.yml b/tasks/install-ohpc.yml similarity index 75% rename from tasks/install.yml rename to tasks/install-ohpc.yml index b7d950d..3cd56cd 100644 --- a/tasks/install.yml +++ b/tasks/install-ohpc.yml @@ -3,16 +3,14 @@ - include_tasks: pre.yml - name: Ensure OpenHPC repos - ansible.builtin.yum_repository: - name: "{{ item.name }}" - description: "{{ item.description | default(omit) }}" - file: "{{ item.file }}" - baseurl: "{{ item.baseurl | default(omit) }}" - metalink: "{{ item.metalink | default(omit) }}" - mirrorlist: "{{ item.mirrorlist | default(omit) }}" - gpgcheck: "{{ item.gpgcheck | default(omit) }}" - gpgkey: "{{ item.gpgkey | default(omit) }}" - loop: "{{ ohpc_repos }}" + ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module] + loop: "{{ ohpc_openhpc_repos[ansible_distribution_major_version] }}" + loop_control: + label: "{{ item.name }}" + +- name: Ensure extra repos + ansible.builtin.yum_repository: "{{ item }}" # noqa: args[module] + loop: "{{ ohpc_extra_repos }}" # NB this gets required ones for OpenHPC too loop_control: label: "{{ item.name }}" diff --git a/tasks/main.yml b/tasks/main.yml index 1ec95d0..e218881 100644 --- a/tasks/main.yml +++ b/tasks/main.yml @@ -2,7 +2,7 @@ - name: Install packages block: - - include_tasks: install.yml + - include_tasks: install-ohpc.yml when: openhpc_enable.runtime | default(false) | bool tags: install diff --git a/tasks/runtime.yml b/tasks/runtime.yml index b346495..c224058 100644 --- a/tasks/runtime.yml +++ b/tasks/runtime.yml @@ -20,12 +20,19 @@ - name: Ensure Slurm directories exists file: - path: "{{ openhpc_state_save_location }}" + path: "{{ item.path }}" owner: slurm group: slurm - mode: 0755 + mode: '0755' state: directory - when: inventory_hostname == openhpc_slurm_control_host + loop: + - path: "{{ openhpc_state_save_location }}" # StateSaveLocation + enable: control + - path: "{{ openhpc_slurm_conf_path | dirname }}" + enable: runtime + - path: "{{ openhpc_slurmd_spool_dir }}" # SlurmdSpoolDir + enable: batch + when: "openhpc_enable[item.enable] | default(false) | bool" - name: Generate a Munge key on control host # NB this is usually a no-op as the package install actually generates a (node-unique) one, so won't usually trigger handler @@ -65,7 +72,7 @@ - name: Template slurmdbd.conf template: src: slurmdbd.conf.j2 - dest: /etc/slurm/slurmdbd.conf + dest: "{{ openhpc_slurm_conf_path | dirname }}/slurmdbd.conf" mode: "0600" owner: slurm group: slurm @@ -82,7 +89,7 @@ - name: Template basic slurm.conf template: - src: slurm.conf.j2 + src: "{{ openhpc_slurm_conf_template }}" dest: "{{ _slurm_conf_tmpfile.path }}" lstrip_blocks: true mode: 0644 @@ -98,6 +105,7 @@ section: '' value: "{{ (item.value | join(',')) if (item.value is sequence and item.value is not string) else item.value }}" no_extra_spaces: true + state: "{{ 'absent' if item.value == '' else 'present' }}" create: no mode: 0644 loop: "{{ openhpc_config | dict2items }}" @@ -109,27 +117,21 @@ - name: Create slurm.conf copy: src: "{{ _slurm_conf_tmpfile.path }}" - dest: /etc/slurm/slurm.conf + dest: "{{ openhpc_slurm_conf_path }}" owner: root group: root mode: 0644 when: openhpc_enable.control | default(false) or not openhpc_slurm_configless - notify: - - Restart slurmctld service + notify: Restart slurmctld service register: ohpc_slurm_conf # NB uses restart rather than reload as number of nodes might have changed -- name: Create gres.conf - template: - src: "{{ openhpc_gres_template }}" - dest: /etc/slurm/gres.conf - mode: "0600" - owner: slurm - group: slurm - when: openhpc_enable.control | default(false) or not openhpc_slurm_configless - notify: - - Restart slurmctld service - register: ohpc_gres_conf +- name: Template other Slurm configuration files + template: "{{ item.template }}" # noqa: risky-file-permissions + loop: "{{ openhpc_config_files }}" + when: "openhpc_enable[item.enable] | default(false) | bool" + notify: Restart slurmctld service + register: ohpc_other_conf # NB uses restart rather than reload as this is needed in some cases - name: Template cgroup.conf @@ -157,7 +159,7 @@ changed_when: true when: - openhpc_slurm_control_host in ansible_play_hosts - - hostvars[openhpc_slurm_control_host].ohpc_slurm_conf.changed or hostvars[openhpc_slurm_control_host].ohpc_gres_conf.changed # noqa no-handler + - hostvars[openhpc_slurm_control_host].ohpc_slurm_conf.changed or hostvars[openhpc_slurm_control_host].ohpc_other_conf.changed # noqa no-handler notify: - Restart slurmd service diff --git a/templates/slurmctld.service.j2 b/templates/slurmctld.service.j2 new file mode 100644 index 0000000..86d73d2 --- /dev/null +++ b/templates/slurmctld.service.j2 @@ -0,0 +1,22 @@ +[Unit] +Description=Slurm controller daemon +After=network-online.target munge.service +Wants=network-online.target +ConditionPathExists={{ openhpc_slurm_conf_path }} + +[Service] +Type=simple +EnvironmentFile=-/etc/sysconfig/slurmctld +EnvironmentFile=-/etc/default/slurmctld +ExecStart={{ openhpc_sbin_dir }}/slurmctld -D -s -f {{ openhpc_slurm_conf_path }} $SLURMCTLD_OPTIONS +ExecReload=/bin/kill -HUP $MAINPID +LimitNOFILE=65536 +TasksMax=infinity + +# Uncomment the following lines to disable logging through journald. +# NOTE: It may be preferable to set these through an override file instead. +#StandardOutput=null +#StandardError=null + +[Install] +WantedBy=multi-user.target diff --git a/templates/slurmd.service.j2 b/templates/slurmd.service.j2 new file mode 100644 index 0000000..501d0e9 --- /dev/null +++ b/templates/slurmd.service.j2 @@ -0,0 +1,25 @@ +[Unit] +Description=Slurm node daemon +After=munge.service network-online.target remote-fs.target +Wants=network-online.target + +[Service] +Type=simple +EnvironmentFile=-/etc/sysconfig/slurmd +EnvironmentFile=-/etc/default/slurmd +ExecStart={{ openhpc_sbin_dir }}/slurmd -D -s $SLURMD_OPTIONS +ExecReload=/bin/kill -HUP $MAINPID +KillMode=process +LimitNOFILE=131072 +LimitMEMLOCK=infinity +LimitSTACK=infinity +Delegate=yes +TasksMax=infinity + +# Uncomment the following lines to disable logging through journald. +# NOTE: It may be preferable to set these through an override file instead. +#StandardOutput=null +#StandardError=null + +[Install] +WantedBy=multi-user.target diff --git a/templates/slurmdbd.service.j2 b/templates/slurmdbd.service.j2 new file mode 100644 index 0000000..591f1d5 --- /dev/null +++ b/templates/slurmdbd.service.j2 @@ -0,0 +1,22 @@ +[Unit] +Description=Slurm DBD accounting daemon +After=network-online.target munge.service mysql.service mysqld.service mariadb.service +Wants=network-online.target +ConditionPathExists={{ openhpc_slurm_conf_path | dirname + '/slurmdbd.conf' }} + +[Service] +Type=simple +EnvironmentFile=-/etc/sysconfig/slurmdbd +EnvironmentFile=-/etc/default/slurmdbd +ExecStart={{ openhpc_sbin_dir }}/slurmdbd -D -s $SLURMDBD_OPTIONS +ExecReload=/bin/kill -HUP $MAINPID +LimitNOFILE=65536 +TasksMax=infinity + +# Uncomment the following lines to disable logging through journald. +# NOTE: It may be preferable to set these through an override file instead. +#StandardOutput=null +#StandardError=null + +[Install] +WantedBy=multi-user.target