The SAT Grafana Dashboards display messages that are generated by the HSN (High Speed Network) and reported through +Redfish. The messages are displayed based on severity.
+Grafana can be accessed via web browser at the following URL:
+
https://sma-grafana.<system_name>.<system_domain>For additional details about how to access the Grafana Dashboards refer to Access the Grafana Monitoring UI in the +SMA product documentation.
+For more information about the interpretation of metrics for the SAT Grafana Dashboards refer to Fabric Telemetry +Kafka Topics in the SMA product documentation.
+There are four Fabric Telemetry dashboards used in SAT that report on the HSN. Two contain chart panels and two display +telemetry in a tabular format.
+| Dashboard Name | +Display Type | +
|---|---|
| Fabric Congestion | +Chart Panels | +
| Fabric RFC3635 | +Chart Panels | +
| Fabric Errors | +Tabular Format | +
| Fabric Port State | +Tabular Format | +
The tabular format presents a single point of telemetry for a given location and metric, either because the telemetry +is not numerical or that it changes infrequently. The value shown is the most recently reported value for that location +during the time range selected, if any. The interval setting is not used for tabular dashboards.
+Shows the Interval and Locations Options for the available telemetry.
+
The value of the Interval option sets the time resolution of the received telemetry. This works a bit like a +histogram, with the available telemetry in an interval of time going into a “bucket” and averaging out to a single +point on the chart or table. The special value auto will choose an interval based on the time range selected.
+For additional information, refer to Grafana Templates and Variables.
+The Locations option allows restriction of the telemetry shown by locations, either individual links or all links +in a switch. The selection presented updates dynamically according to time range, except for the errors dashboard, +which always has entries for all links and switches, although the errors shown are restricted to the selected time +range.
+The chart panels for the RFC3635 and Congestion dashboards allow selection of a single location from the chart’s legend +or the trace on the chart.
+
SAT Grafana Dashboards provide system administrators a way to view fabric telemetry data across all Rosetta switches in +the system and assess the past and present health of the high-speed network. It also allows the ability to drill down +to view data for specific ports on specific switches.
+This dashboard contains the variable, Port Type not found in the other dashboards. The possible values are edge, +local, and global and correspond to the link’s relationship to the network topology. The locations presented in the +panels are restricted to the values (any combination, defaults to “all”) selected.
+The metric values for links of a given port type are similar in value to each other but very distinct from the values of +other types. If the values for different port types are all plotted together, the values for links with lower values are +indistinguishable from zero when plotted.
+The port type of a link is reported as a port state “subtype” event when defined at port initialization.
+
This dashboard reports error counters in a tabular format in three panels.
+There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+Unlike other dashboards, the locations presented are all locations in the system rather than having telemetry within +the time range selected. However, the values are taken from telemetry within the time range.
+
There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+The Fabric Port State telemetry is distinct because it typically is not numeric. It also updates infrequently, so a +long time range may be necessary to obtain any values. Port State is refreshed daily, so a time range of 24 hours +results in all states for all links in the system being shown.
+The three columns named, group, switch, and port are not port state events, but extra information included with +all port state events.
+
For additional information on performance counters, refer to +Definitions of Managed Objects for the Ethernet-like Interface Types, +an Internet standards document.
+Because these metrics are counters that only increase over time, the values plotted are the change in the counter’s +value over the interval setting.
+ + + + + +Kibana is an open source analytics and visualization platform designed to search, view, and interact with data stored +in Elasticsearch indices. Kibana runs as a web service and has a browser-based interface. It offers visual output of +node data in the forms of charts, tables and maps that display real-time Elasticsearch queries. Viewing system data in +this way breaks down the complexity of large data volumes into easily understood information.
+Kibana can be accessed via web browser at the following URL:
+https://sma-kibana.<system_name>.<system_domain>For additional details about how to access the Kibana Dashboards refer to View Logs Via Kibana in the SMA product +documentation.
+Additional details about the AER, ATOM, Heartbeat, Kernel, MCE, and Rasdaemon Kibana Dashboards are included in this +table.
+| Dashboard | +Short Description | +Long Description | +Kibana Visualization and Search Name | +
|---|---|---|---|
| sat-aer | +AER corrected | +Corrected Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-corrected Search: sat-aer-corrected | +
| sat-aer | +AER fatal | +Fatal Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-fatal Search: sat-aer-fatal | +
| sat-atom | +ATOM failures | +Application Task Orchestration and Management tests are run on a node when a job finishes. Test failures are logged. | +sat-atom-failed | +
| sat-atom | +ATOM admindown | +Application Task Orchestration and Management test failures can result in nodes being marked admindown. An admindown node is not available for job launch. | +sat-atom-admindown | +
| sat-heartbeat | +Heartbeat loss events | +Heartbeat loss event messages reported by the hbtd pods that monitor for heartbeats across nodes in the system. | +sat-heartbeat | +
| sat-kernel | +Kernel assertions | +The kernel software performs a failed assertion when some condition represents a serious fault. The node goes down. | +sat-kassertions | +
| sat-kernel | +Kernel panics | +The kernel panics when something is seriously wrong. The node goes down. | +sat-kernel-panic | +
| sat-kernel | +Lustre bugs (LBUGs) | +The Lustre software in the kernel stack performs a failed assertion when some condition related to file system logic represents a serious fault. The node goes down. | +sat-lbug | +
| sat-kernel | +CPU stalls | +CPU stalls are serous conditions that can reduce node performance, and sometimes cause a node to go down. Technically these are Read-Copy-Update stalls where software in the kernel stack holds onto memory for too long. Read-Copy-Update is a vital aspect of kernel performance and rather esoteric. | +sat-cpu-stall | +
| sat-kernel | +Out of memory | +An Out Of Memory (OOM) condition has occurred. The kernel must kill a process to continue. The kernel will select an expendable process when possible. If there is no expendable process the node usually goes down in some manner. Even if there are expendable processes the job is likely to be impacted. OOM conditions are best avoided. | +sat-oom | +
| sat-mce | +MCE | +Machine Check Exceptions (MCE) are errors detected at the processor level. | +sat-mce | +
| sat-rasdaemon | +rasdaemon errors | +Errors from the rasdaemon service on nodes. The rasdaemon service is the Reliability, Availability, and Serviceability Daemon, and it is intended to collect all hardware error events reported by the linux kernel, including PCI and MCE errors. This may include certain HSN errors in the future. |
+sat-rasdaemon-error | +
| sat-rasdaemon | +rasdaemon messages | +All messages from the rasdaemon service on nodes. |
+sat-rasdaemon | +
By default, search highlighting is enabled. This procedure instructs how to disable search highlighting.
+The Kibana Dashboard should be open on your system.
+Navigate to Management
+Navigate to Advanced Settings in the Kibana section, below the Elastic search section
+Scroll down to the Discover section
+Change Highlight results from on to off
+Click Save to save changes
+The AER Dashboard displays errors that come from the PCI Express Advanced Error Reporting (AER) driver. These errors +are split up into separate visualizations depending on whether they are fatal or corrected errors.
+Go to the dashboard section.
+Select sat-aer dashboard.
+Choose the time range of interest.
+View the Corrected and Fatal Advanced Error Reporting messages from PCI Express devices on each node. View the +matching log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on +the left. If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass +next to each NID.
+The ATOM (Application Task Orchestration and Management) Dashboard displays node failures that occur during health +checks and application test failures. Some test failures are of possible interest even though a node is not marked +admindown or otherwise fails. They are of clear interest if a node is marked admindown, and might provide +clues if a node otherwise fails. They might also show application problems.
+HPE Cray EX is installed on the system along with the System Admin Toolkit, which contains the ATOM Kibana Dashboard.
+Go to the dashboard section.
+Select sat-atom dashboard.
+Choose the time range of interest.
+View any nodes marked admindown and any ATOM test failures. These failures occur during health checks and +application test failures. Test failures marked admindown are important to note. View the matching log messages +in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. If desired, +results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Heartbeat Dashboard displays heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods +are responsible for monitoring nodes in the system for heartbeat loss.
+Go to the dashboard section.
+Select sat-heartbeat dashboard.
+Choose the time range of interest.
+View the heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods are responsible for +monitoring nodes in the system for heartbeat loss.View the matching log messages in the panel.
+The Kernel Dashboard displays compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. +The messages reveal if Lustre has experienced a fatal error on any compute nodes in the system. A CPU stall is a serious +problem that might result in a node failure. Out-of-memory conditions can be due to applications or system problems and +may require expert analysis. They provide useful clues for some node failures and may reveal if an application is using +too much memory.
+Go to the dashboard section.
+Select sat-kernel dashboard.
+Choose the time range of interest.
+View the compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. View the matching +log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. +If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to +each NID.
+The MCE Dashboard displays CPU detected processor-level hardware errors.
+Go to the dashboard section.
+Select sat-mce dashboard.
+Choose the time range of interest.
+View the Machine Check Exceptions (MCEs) listed including the counts per NID (node). For an MCE, the CPU number and +DIMM number can be found in the message, if applicable. View the matching log messages in the panel(s) on the right, +and view the counts of each message per NID in the panel(s) on the left. If desired, results can be filtered by NID +by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Rasdaemon Dashboard displays errors that come from the Reliability, Availability, and Serviceability (RAS) daemon
+service on nodes in the system. This service collects all hardware error events reported by the linux kernel, including
+PCI and MCE errors. As a result there may be some duplication between the messages presented here and the messages
+presented in the MCE and AER dashboards. This dashboard splits up the messages into two separate visualizations, one
+for only messages of severity “emerg” or “err” and another for all messages from rasdaemon.
Go to the dashboard section.
+Select sat-rasdaemon dashboard.
+Choose the time range of interest.
+View the errors that come from the Reliability, Availability, and Serviceability (RAS) daemon service on nodes in +the system. View the matching log messages in the panel(s) on the right, and view the counts of each message per NID +in the panel(s) on the left. If desired, results can be filtered by NID by clicking the icon showing a + inside +a magnifying glass next to each NID.
+Describes how to install the System Admin Toolkit (SAT) product stream.
+...) in shell output indicate omitted lines.2.1.x with the version of the SAT product stream
+being installed.Start a typescript.
+The typescript will record the commands and the output from this installation.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Copy the release distribution gzipped tar file to ncn-m001.
Unzip and extract the release distribution, 2.1.x.
ncn-m001# tar -xvzf sat-2.1.x.tar.gz
+Change directory to the extracted release distribution directory.
+ncn-m001# cd sat-2.1.x
+Run the installer: install.sh.
+The script produces a lot of output. The last several lines are included +below for reference.
+ncn-m001# ./install.sh
+...
+ConfigMap data updates exist; Exiting.
++ clean-install-deps
++ for image in "${vendor_images[@]}"
++ podman rmi -f docker.io/library/cray-nexus-setup:sat-2.1.x-20210804163905-8dbb87d
+Untagged: docker.io/library/cray-nexus-setup:sat-2.1.x-20210804163905-8dbb87d
+Deleted: 2c196c0c6364d9a1699d83dc98550880dc491cc3433a015d35f6cab1987dd6da
++ for image in "${vendor_images[@]}"
++ podman rmi -f docker.io/library/skopeo:sat-2.1.x-20210804163905-8dbb87d
+Untagged: docker.io/library/skopeo:sat-2.1.x-20210804163905-8dbb87d
+Deleted: 1b38b7600f146503e246e753cd9df801e18409a176b3dbb07b0564e6bc27144c
+Check the return code of the installer. Zero indicates a successful installation.
+ncn-m001# echo $?
+0
+Check the progress of the SAT configuration import Kubernetes job, which is
+initiated by install.sh.
If the “Pods Statuses” appear as “Succeeded”, the job has completed +successfully. The job usually takes between 30 seconds and 2 minutes.
+ncn-m001# kubectl describe job sat-config-import-2.1.x -n services
+...
+Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
+...
+The job’s progress may be monitored using kubectl logs. The example below includes
+the final log lines from a successful configuration import Kubernetes job.
ncn-m001# kubectl logs -f -n services --selector \
+ job-name=sat-config-import-2.1.x --all-containers
+...
+ConfigMap update attempt=1
+Resting 1s before reading ConfigMap
+ConfigMap data updates exist; Exiting.
+2021-08-04T21:50:10.275886Z info Agent has successfully terminated
+2021-08-04T21:50:10.276118Z warning envoy main caught SIGTERM
+# Completed on Wed Aug 4 21:49:44 2021
+The following error may appear in this log, but it can be ignored.
+error accept tcp [::]:15020: use of closed network connection
+Optional: Remove the SAT release distribution tar file and extracted directory.
+ncn-m001# rm sat-2.2.x.tar.gz
+ncn-m001# rm -rf sat-2.2.x/
+Upgrade only: Ensure that the environment variable SAT_TAG is not set
+in the ~/.bashrc file on any of the management NCNs.
NOTE: This step should only be required when updating from +Shasta 1.4.1 or Shasta 1.4.2.
+The following example assumes three manager NCNs: ncn-m001, ncn-m002, and ncn-m003,
+and shows output from a system in which no further action is needed.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m003: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+The following example shows that SAT_TAG is set in ~/.bashrc on ncn-m002.
+Remove that line from the ~/.bashrc file on ncn-m002.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+ncn-m002: export SAT_TAG=3.5.0
+ncn-m003: source <(kubectl completion bash)
+Stop the typescript.
+NOTE: This step can be skipped if you wish to use the same typescript +for the remainder of the SAT install. See Next Steps.
+ncn-m001# exit
+SAT version 2.1.x is now installed/upgraded, meaning the SAT 2.1.x release
+has been loaded into the system software repository.
sat command won’t be available until the NCN Personalization
+procedure has been executed.If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the sections listed below.
+NOTE: The NCN Personalization procedure is required when +upgrading SAT. The setup procedures in SAT Setup, however, are +not required when upgrading SAT. They should have been executed +during the first installation of SAT.
+Execute the NCN Personalization procedure:
+ +If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the upgrade procedures:
+ +Describes how to perform NCN personalization using CFS. This personalization process +will configure the System Admin Toolkit (SAT) product stream.
+...) in shell output indicate omitted lines.2.1.x with the version of the SAT product stream
+being installed.Start a typescript if not already using one.
+The typescript will capture the commands and the output from this installation procedure.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Get the git commit ID for the branch with a version number matching the version of SAT.
+This represents a revision of Ansible configuration content stored in VCS.
+Get and store the VCS password (required to access the remote VCS repo).
+ncn-m001# VCS_PASS=$(kubectl get secret -n services vcs-user-credentials \
+ --template={{.data.vcs_password}} | base64 --decode)
+In this example, the git commit ID is 82537e59c24dd5607d5f5d6f92cdff971bd9c615,
+and the version number is 2.1.x.
ncn-m001# git ls-remote \
+ https://crayvcs:$VCS_PASS@api-gw-service-nmn.local/vcs/cray/sat-config-management.git \
+ refs/heads/cray/sat/*
+...
+82537e59c24dd5607d5f5d6f92cdff971bd9c615 refs/heads/cray/sat/2.1.x
+Add a sat layer to the CFS configuration(s) associated with the manager NCNs.
Get the name(s) of the CFS configuration(s).
+NOTE: Each manager NCN uses a single CFS configuration. An individual CFS configuration +may be used by any number of manage NCNs, i.e., three manager NCNs might use one, +two, or three CFS configurations.
+In the following example, all three manager NCNs use the same CFS configuration – ncn-personalization.
ncn-m001:~ # for component in $(cray hsm state components list \
+ --role Management --subrole Master --format json | jq -r \
+ '.Components | .[].ID'); do cray cfs components describe $component \
+ --format json | jq -r '.desiredConfig'; done
+ncn-personalization
+ncn-personalization
+ncn-personalization
+In the following example, the three manager NCNs all use different configurations, +each with a unique name.
+ncn-personalization-m001
+ncn-personalization-m002
+ncn-personalization-m003
+Execute the following sub-steps (3.2 through 3.5) once for each unique CFS +configuration name.
+NOTE: Examples in the following sub-steps assume that all manager NCNs use the
+CFS configuration ncn-personalization.
Get the current configuration layers for each CFS configuration, and save the +data to a local JSON file.
+The JSON file created in this sub-step will serve as a template for updating +an existing CFS configuration, or creating a new one.
+ncn-m001# cray cfs configurations describe ncn-personalization --format \
+ json | jq '{ layers }' > ncn-personalization.json
+If the configuration does not exist yet, you may see the following error.
+In this case, create a new JSON file for that CFS configuration, e.g., ncn-personalization.json.
Error: Configuration could not found.: Configuration ncn-personalization could not be found
+NOTE: For more on CFS configuration management, refer to “Manage a Configuration +with CFS” in the CSM product documentation.
+Append a sat layer to the end of the JSON file’s list of layers.
If the file already contains a sat layer entry, update it.
If the configuration data could not be found in the previous sub-step, the JSON file
+will be empty. In this case, copy the ncn-personalization.json example below,
+paste it into the JSON file, delete the ellipsis, and make appropriate changes to
+the sat layer entry.
Use the git commit ID from step 8, e.g. 82537e59c24dd5607d5f5d6f92cdff971bd9c615.
NOTE: The name value in the example below may be changed, but the installation
+procedure uses the example value, sat-ncn. If an alternate value is used, some
+of the following examples must be updated accordingly before they are executed.
ncn-m001# vim ncn-personalization.json
+...
+ncn-m001# cat ncn-personalization.json
+{
+ "layers": [
+ ...
+ {
+ "cloneUrl": "https://api-gw-service-nmn.local/vcs/cray/sat-config-management.git",
+ "commit": "82537e59c24dd5607d5f5d6f92cdff971bd9c615",
+ "name": "sat-ncn",
+ "playbook": "sat-ncn.yml"
+ }
+ ]
+}
+Update the existing CFS configuration, or create a new one.
+The command should output a JSON-formatted representation of the CFS configuration,
+which will look like the JSON file, but with lastUpdated and name fields.
ncn-m001# cray cfs configurations update ncn-personalization --file \
+ ncn-personalization.json --format json
+{
+ "lastUpdated": "2021-08-05T16:38:53Z",
+ "layers": {
+ ...
+ },
+ "name": "ncn-personalization"
+}
+Optional: Delete the JSON file.
+NOTE: There is no reason to keep the file. If you keep it, verify that +it is up-to-date with the actual CFS configuration before using it again.
+ncn-m001# rm ncn-personalization.json
+Invoke the CFS configurations that you created or updated in the previous step.
+This step will create a CFS session based on the given configuration and install +SAT on the associated manager NCNs.
+The --configuration-limit option causes only the sat-ncn layer of the configuration,
+ncn-personalization, to run.
CAUTION: In this example, the session --name is sat-session. That value
+is only an example. Declare a unique name for each configuration session.
You should see a representation of the CFS session in the output.
+ncn-m001# cray cfs sessions create --name sat-session --configuration-name \
+ ncn-personalization --configuration-limit sat-ncn
+name="sat-session"
+
+[ansible]
+...
+Execute this step once for each unique CFS configuration that you created or +updated in the previous step.
+Monitor the progress of each CFS session.
+First, list all containers associated with the CFS session:
+ncn-m001# kubectl get pod -n services --selector=cfsession=sat-session \
+ -o json | jq '.items[0].spec.containers[] | .name'
+"inventory"
+"ansible-1"
+"istio-proxy"
+Next, get the logs for the ansible-1 container.
NOTE: the trailing digit might differ from “1”. It is the zero-based
+index of the sat-ncn layer within the configuration’s layers.
ncn-m001# kubectl logs -c ansible-1 --tail 100 -f -n services \
+ --selector=cfsession=sat-session
+Ansible plays, which are run by the CFS session, will install SAT on all the +manager NCNs on the system. Successful results for all of the manager NCN xnames +can be found at the end of the container log. For example:
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Execute this step for each unique CFS configuration.
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify that SAT was successfully configured.
+If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
NOTE: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
ncn-m001# sat --version
+sat 3.7.0
+NOTE: Upon first running sat, you may see additional output while the sat
+container image is downloaded. This will occur the first time sat is run on
+each manager NCN. For example, if you run sat for the first time on ncn-m001
+and then for the first time on ncn-m002, you will see this additional output
+both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+Stop the typescript.
+ncn-m001# exit
+SAT version 2.1.x is now configured:
If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the remaining SAT Setup or SAT Post-Upgrade procedures.
+If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the SAT Post-Upgrade procedures:
+ +Initially, as part of the installation and configuration, SAT authentication is set up so sat commands can be used in
+later steps of the install process. The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For instructions on editing Role Mappings see
+Create Internal User Accounts in the Keycloak Shasta Realm in the CSM product documentation.
+For additional information on SAT authentication, see System Security and Authentication in the CSM
+documentation.
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Some SAT subcommands make requests to the Shasta services through the API gateway and thus require authentication to
+the API gateway in order to function. Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured (see: Generate SAT S3 Credentials). In order to use the SAT S3 bucket,
+the System Administrator must generate the S3 access key and secret keys and write them to a local file. This must be
+done on every Kubernetes manager node where SAT commands are run.
Below is a table describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bootsys |
+Requires authentication to the API gateway. Requires kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat k8s |
+Requires kubernetes configuration and authentication, which is automatically configured on ncn-w001 during the install. | +sat-k8s |
+Report on kubernetes replicasets that have co-located replicas (i.e. replicas on the same node). | +
sat linkhealth |
++ | + | This command has been deprecated. | +
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node xnames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC xnames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, you must run the sat auth command. This command will prompt for a password
+on the command line. The username value is obtained from the following locations, in order of higher precedence to lower
+precedence:
--username global command-line option.username option in the api_gateway section of the config file at ~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file will be obtained and saved to
+~/.config/sat/tokens. Subsequent sat commands will determine the username the same way as sat auth described above,
+and will use the token for that username if it has been obtained and saved by sat auth.
sat CLI has been installed following Install The System Admin Toolkit Product Stream.The following is the procedure to globally configure the username used by SAT and authenticate to the API gateway:
+Generate a default SAT configuration file, if one does not exist.
+ncn-m001# sat init
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the config file already exists, it will print out an error:
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway section of the config file. E.g.:
username = "crayadmin"
+Run sat auth. Enter your password when prompted. E.g.:
ncn-m001# sat auth
+Password for crayadmin:
+Succeeded!
+Other sat commands are now authenticated to make requests to the API gateway. E.g.:
ncn-m001# sat status
+Generate S3 credentials and write them to a local file so the SAT user can access S3 storage. In order to use the SAT +S3 bucket, the System Administrator must generate the S3 access key and secret keys and write them to a local file. +This must be done on every Kubernetes master node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the site-specific information set with sat setrev
+(see: Run Sat Setrev to Set System Information).
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+sat CLI has been installed following Install The System Admin Toolkit Product Stream.sat configuration file has been created (See SAT Authentication).Ensure the files are readable only by root.
ncn-m001# touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+ncn-m001# chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+Get the SAT configuration file’s endpoint valie.
+NOTE: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
ncn-m001# grep endpoint ~/.config/sat/sat.toml
+# endpoint = "https://rgw-vip.nmn"
+Get the sat-s3-credentials secret’s endpoint value.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, modify the SAT configuration file’s endpoint value to match the secret’s.
+Copy SAT configurations to every manager node on the system.
+ncn-m001# for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+NOTE: Depending on how many manager nodes are on the system, the list of manager nodes may +be different. This example assumes three manager nodes, where the configuration files must be +copied from ncn-m001 to ncn-m002 and ncn-m003. Therefore, the list of hosts above is ncn-m002 +and ncn-m003.
+NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Run sat setrev to set System Revision Information. Follow the on-screen prompts.
ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Run sat showrev to verify System Revision Information. The following tables contain example information.
ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | Shasta |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+After upgrading from a previous version of SAT, the old version of the cray/cray-sat
+container image will remain in the registry on the system. It is not removed
+automatically, but it will not be the default version.
The admin can remove the older version of the cray/cray-sat container image.
The cray-product-catalog Kubernetes configuration map will also show all versions
+of SAT that are installed. The command sat showrev --products will display these
+versions. See the example:
ncn-m001# sat showrev --products
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------------------+-----------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+--------------------+-----------------------+
+...
+| sat | 2.1.3 | - | - |
+| sat | 2.0.4 | - | - |
+...
++--------------+-----------------+--------------------+-----------------------+
+After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
+In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+ + + + + +The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands +used on the Cray XC platform. For more information on SAT commands, see System Admin Toolkit Command Overview.
+Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
+Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and +are reported through Redfish.
+SAT is installed as a separate product as part of the HPE Cray EX System base installation.
+Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the +HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are +similarities between SAT commands and xt commands used on the Cray XC platform.
+The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using podman, a daemonless container runtime. SAT runs on Kubernetes
+manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
+sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
+(CONTAINER-ID)sat-container# sat status
+ncn-m001# sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
+example.
ncn-m001# sat-man status
+A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
+ncn-m001# man sat-podman
+The host name in a command prompt indicates where the command must be run. The account that must run the command is +also indicated in the prompt.
+root or super-user account always has the # character at the end of the prompt and has the host name of the
+host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
+referred to as user.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run on one of the Kubernetes Manager servers. (Non-interactive) | +
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
Examples of the sat status command used by an administrator:
ncn-m001# sat status
+ncn-m001# sat bash
+(CONTAINER_ID) sat-container# sat status
+Most sat subcommands depend on services or components from other products in the
+HPE Cray EX (Shasta) software stack. The following list shows these dependencies
+for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bootsyssat diagsat firmwaresat hwinvsat hwmatchsat initNone
+sat k8ssat nid2xnamesat sensorssat setrevsat showrevsat statussat swapsat switchDeprecated: See sat swap
sat xname2nidWe released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named “ncn-personalization”.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+executables on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and config file options +have been removed. For further instructions, see Remove Obsolete Configuration +File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes masters and workers.
sat firmware.cray-sat container image.sat firmware command.We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes management cluster (i.e., workers
+and masters). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Config File Location ChangeThe default location of the SAT config file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own config files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment variable.
Additionally, if a config file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies option.--list-node-accels option. The count of
+node accelerators is also included for each node.--list-node-accel-risers
+option. The count of node accelerator risers is also included for each node.--list-node-hsn-nics
+option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For detailed +instructions, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding config-file
+option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding config file options were
+deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchSee the System Admin Toolkit Command Overview +and the table of commands in the SAT Authentication section +of this document for more details on each of these commands.
+ + + + + +The SAT Grafana Dashboards display messages that are generated by the HSN (High Speed Network) and reported through +Redfish. The messages are displayed based on severity.
+Grafana can be accessed via web browser at the following URL:
+https://sma-grafana.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Grafana being https://sma-grafana.EXAMPLE_DOMAIN.com
For additional details about how to access the Grafana Dashboards refer to Access the Grafana Monitoring UI in the +SMA product documentation.
+For more information about the interpretation of metrics for the SAT Grafana Dashboards refer to Fabric Telemetry +Kafka Topics in the SMA product documentation.
+There are four Fabric Telemetry dashboards used in SAT that report on the HSN. Two contain chart panels and two display +telemetry in a tabular format.
+| Dashboard Name | +Display Type | +
|---|---|
| Fabric Congestion | +Chart Panels | +
| Fabric RFC3635 | +Chart Panels | +
| Fabric Errors | +Tabular Format | +
| Fabric Port State | +Tabular Format | +
The tabular format presents a single point of telemetry for a given location and metric, either because the telemetry +is not numerical or that it changes infrequently. The value shown is the most recently reported value for that location +during the time range selected, if any. The interval setting is not used for tabular dashboards.
+Shows the Interval and Locations Options for the available telemetry.
+
The value of the Interval option sets the time resolution of the received telemetry. This works a bit like a +histogram, with the available telemetry in an interval of time going into a “bucket” and averaging out to a single +point on the chart or table. The special value auto will choose an interval based on the time range selected.
+For additional information, refer to Grafana Templates and Variables.
+The Locations option allows restriction of the telemetry shown by locations, either individual links or all links +in a switch. The selection presented updates dynamically according to time range, except for the errors dashboard, +which always has entries for all links and switches, although the errors shown are restricted to the selected time +range.
+The chart panels for the RFC3635 and Congestion dashboards allow selection of a single location from the chart’s legend +or the trace on the chart.
+
SAT Grafana Dashboards provide system administrators a way to view fabric telemetry data across all Rosetta switches in +the system and assess the past and present health of the high-speed network. It also allows the ability to drill down +to view data for specific ports on specific switches.
+This dashboard contains the variable, Port Type not found in the other dashboards. The possible values are edge, +local, and global and correspond to the link’s relationship to the network topology. The locations presented in the +panels are restricted to the values (any combination, defaults to “all”) selected.
+The metric values for links of a given port type are similar in value to each other but very distinct from the values of +other types. If the values for different port types are all plotted together, the values for links with lower values are +indistinguishable from zero when plotted.
+The port type of a link is reported as a port state “subtype” event when defined at port initialization.
+
This dashboard reports error counters in a tabular format in three panels.
+There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+Unlike other dashboards, the locations presented are all locations in the system rather than having telemetry within +the time range selected. However, the values are taken from telemetry within the time range.
+
There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+The Fabric Port State telemetry is distinct because it typically is not numeric. It also updates infrequently, so a +long time range may be necessary to obtain any values. Port State is refreshed daily, so a time range of 24 hours +results in all states for all links in the system being shown.
+The three columns named, group, switch, and port are not port state events, but extra information included with +all port state events.
+
For additional information on performance counters, refer to +Definitions of Managed Objects for the Ethernet-like Interface Types, +an Internet standards document.
+Because these metrics are counters that only increase over time, the values plotted are the change in the counter’s +value over the interval setting.
+ + + + + +Kibana is an open source analytics and visualization platform designed to search, view, and interact with data stored +in Elasticsearch indices. Kibana runs as a web service and has a browser-based interface. It offers visual output of +node data in the forms of charts, tables and maps that display real-time Elasticsearch queries. Viewing system data in +this way breaks down the complexity of large data volumes into easily understood information.
+Kibana can be accessed via web browser at the following URL:
+https://sma-kibana.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Kibana being https://sma-kibana.EXAMPLE_DOMAIN.com
For additional details about how to access the Kibana Dashboards refer to View Logs Via Kibana in the SMA product +documentation.
+Additional details about the AER, ATOM, Heartbeat, Kernel, MCE, and Rasdaemon Kibana Dashboards are included in this +table.
+| Dashboard | +Short Description | +Long Description | +Kibana Visualization and Search Name | +
|---|---|---|---|
| sat-aer | +AER corrected | +Corrected Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-corrected Search: sat-aer-corrected | +
| sat-aer | +AER fatal | +Fatal Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-fatal Search: sat-aer-fatal | +
| sat-atom | +ATOM failures | +Application Task Orchestration and Management tests are run on a node when a job finishes. Test failures are logged. | +sat-atom-failed | +
| sat-atom | +ATOM admindown | +Application Task Orchestration and Management test failures can result in nodes being marked admindown. An admindown node is not available for job launch. | +sat-atom-admindown | +
| sat-heartbeat | +Heartbeat loss events | +Heartbeat loss event messages reported by the hbtd pods that monitor for heartbeats across nodes in the system. | +sat-heartbeat | +
| sat-kernel | +Kernel assertions | +The kernel software performs a failed assertion when some condition represents a serious fault. The node goes down. | +sat-kassertions | +
| sat-kernel | +Kernel panics | +The kernel panics when something is seriously wrong. The node goes down. | +sat-kernel-panic | +
| sat-kernel | +Lustre bugs (LBUGs) | +The Lustre software in the kernel stack performs a failed assertion when some condition related to file system logic represents a serious fault. The node goes down. | +sat-lbug | +
| sat-kernel | +CPU stalls | +CPU stalls are serous conditions that can reduce node performance, and sometimes cause a node to go down. Technically these are Read-Copy-Update stalls where software in the kernel stack holds onto memory for too long. Read-Copy-Update is a vital aspect of kernel performance and rather esoteric. | +sat-cpu-stall | +
| sat-kernel | +Out of memory | +An Out Of Memory (OOM) condition has occurred. The kernel must kill a process to continue. The kernel will select an expendable process when possible. If there is no expendable process the node usually goes down in some manner. Even if there are expendable processes the job is likely to be impacted. OOM conditions are best avoided. | +sat-oom | +
| sat-mce | +MCE | +Machine Check Exceptions (MCE) are errors detected at the processor level. | +sat-mce | +
| sat-rasdaemon | +rasdaemon errors | +Errors from the rasdaemon service on nodes. The rasdaemon service is the Reliability, Availability, and Serviceability Daemon, and it is intended to collect all hardware error events reported by the linux kernel, including PCI and MCE errors. This may include certain HSN errors in the future. |
+sat-rasdaemon-error | +
| sat-rasdaemon | +rasdaemon messages | +All messages from the rasdaemon service on nodes. |
+sat-rasdaemon | +
By default, search highlighting is enabled. This procedure instructs how to disable search highlighting.
+The Kibana Dashboard should be open on your system.
+Navigate to Management
+Navigate to Advanced Settings in the Kibana section, below the Elastic search section
+Scroll down to the Discover section
+Change Highlight results from on to off
+Click Save to save changes
+The AER Dashboard displays errors that come from the PCI Express Advanced Error Reporting (AER) driver. These errors +are split up into separate visualizations depending on whether they are fatal or corrected errors.
+Go to the dashboard section.
+Select sat-aer dashboard.
+Choose the time range of interest.
+View the Corrected and Fatal Advanced Error Reporting messages from PCI Express devices on each node. View the +matching log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on +the left. If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass +next to each NID.
+The ATOM (Application Task Orchestration and Management) Dashboard displays node failures that occur during health +checks and application test failures. Some test failures are of possible interest even though a node is not marked +admindown or otherwise fails. They are of clear interest if a node is marked admindown, and might provide +clues if a node otherwise fails. They might also show application problems.
+HPE Cray EX is installed on the system along with the System Admin Toolkit, which contains the ATOM Kibana Dashboard.
+Go to the dashboard section.
+Select sat-atom dashboard.
+Choose the time range of interest.
+View any nodes marked admindown and any ATOM test failures. These failures occur during health checks and +application test failures. Test failures marked admindown are important to note. View the matching log messages +in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. If desired, +results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Heartbeat Dashboard displays heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods +are responsible for monitoring nodes in the system for heartbeat loss.
+Go to the dashboard section.
+Select sat-heartbeat dashboard.
+Choose the time range of interest.
+View the heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods are responsible for +monitoring nodes in the system for heartbeat loss.View the matching log messages in the panel.
+The Kernel Dashboard displays compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. +The messages reveal if Lustre has experienced a fatal error on any compute nodes in the system. A CPU stall is a serious +problem that might result in a node failure. Out-of-memory conditions can be due to applications or system problems and +may require expert analysis. They provide useful clues for some node failures and may reveal if an application is using +too much memory.
+Go to the dashboard section.
+Select sat-kernel dashboard.
+Choose the time range of interest.
+View the compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. View the matching +log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. +If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to +each NID.
+The MCE Dashboard displays CPU detected processor-level hardware errors.
+Go to the dashboard section.
+Select sat-mce dashboard.
+Choose the time range of interest.
+View the Machine Check Exceptions (MCEs) listed including the counts per NID (node). For an MCE, the CPU number and +DIMM number can be found in the message, if applicable. View the matching log messages in the panel(s) on the right, +and view the counts of each message per NID in the panel(s) on the left. If desired, results can be filtered by NID +by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Rasdaemon Dashboard displays errors that come from the Reliability, Availability, and Serviceability (RAS) daemon
+service on nodes in the system. This service collects all hardware error events reported by the linux kernel, including
+PCI and MCE errors. As a result there may be some duplication between the messages presented here and the messages
+presented in the MCE and AER dashboards. This dashboard splits up the messages into two separate visualizations, one
+for only messages of severity “emerg” or “err” and another for all messages from rasdaemon.
Go to the dashboard section.
+Select sat-rasdaemon dashboard.
+Choose the time range of interest.
+View the errors that come from the Reliability, Availability, and Serviceability (RAS) daemon service on nodes in +the system. View the matching log messages in the panel(s) on the right, and view the counts of each message per NID +in the panel(s) on the left. If desired, results can be filtered by NID by clicking the icon showing a + inside +a magnifying glass next to each NID.
+Describes how to install the System Admin Toolkit (SAT) product stream.
+...) in shell output indicate omitted lines.2.2.x with the version of the SAT product stream
+being installed.Start a typescript.
+The typescript will record the commands and the output from this installation.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Copy the release distribution gzipped tar file to ncn-m001.
Unzip and extract the release distribution, 2.2.x.
ncn-m001# tar -xvzf sat-2.2.x.tar.gz
+Change directory to the extracted release distribution directory.
+ncn-m001# cd sat-2.2.x
+Run the installer: install.sh.
+The script produces a lot of output. A successful install ends with “SAT +version 2.2.x has been installed”.
+ncn-m001# ./install.sh
+...
+====> Updating active CFS configurations
+...
+====> SAT version 2.2.x has been installed.
+Upgrade only: Record the names of the CFS configuration or
+configurations modified by install.sh.
The install.sh script attempts to modify any CFS configurations that apply
+to the master management NCNs. During an upgrade, install.sh will log
+messages indicating the CFS configuration or configurations that were
+modified. For example, if there are three master nodes all using the same
+CFS configuration named “ncn-personalization”, the output would look like
+this:
====> Updating active CFS configurations
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, x3000c0s3b0n0, x3000c0s5b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s1b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s3b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s5b0n0
+INFO: Updating CFS configuration "ncn-personalization"
+INFO: Updating existing layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml in configuration "ncn-personalization".
+INFO: Key "name" in layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml updated from sat-ncn to sat-2.2.16
+INFO: Successfully updated layers in configuration "ncn-personalization"
+Save the name of each CFS configuration updated by the installer. In the +previous example, a single configuration named “ncn-personalization” was +updated, so that name is saved to a temporary file.
+ncn-m001# echo ncn-personalization >> /tmp/sat-ncn-cfs-configurations.txt
+Repeat the previous command for each CFS configuration that was updated.
+Upgrade only: Save the new name of the SAT CFS configuration layer.
+In the example install.sh output above, the new layer name is
+sat-2.2.16. Save this value to a file to be used later.
ncn-m001# echo sat-2.2.16 > /tmp/sat-layer-name.txt
+Fresh install only: Save the CFS configuration layer for SAT to a file +for later use.
+The install.sh script attempts to modify any CFS configurations that apply
+to the master management NCNs. During a fresh install, no such CFS
+configurations will be found, and it will instead log the SAT configuration
+layer that must be added to the CFS configuration that will be created. Here
+is an example of the output in that case:
====> Updating active CFS configurations
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, x3000c0s3b0n0, x3000c0s5b0n0
+WARNING: No CFS configurations found that apply to components with role Management and subrole Master.
+INFO: The following sat layer should be used in the CFS configuration that will be applied to NCNs with role Management and subrole Master.
+{
+ "name": "sat-2.2.15",
+ "commit": "9a74b8f5ba499af6fbcecfd2518a40e081312933",
+ "cloneUrl": "https://api-gw-service-nmn.local/vcs/cray/sat-config-management.git",
+ "playbook": "sat-ncn.yml"
+}
+Save the JSON output to a file for later use. For example:
+ncn-m001# cat > /tmp/sat-layer.json <<EOF
+> {
+> "name": "sat-2.2.15",
+> "commit": "9a74b8f5ba499af6fbcecfd2518a40e081312933",
+> "cloneUrl": "https://api-gw-service-nmn.local/vcs/cray/sat-config-management.git",
+> "playbook": "sat-ncn.yml"
+> }
+> EOF
+Do not copy the previous command verbatim. Use the JSON output from the
+install.sh script.
Optional: Remove the SAT release distribution tar file and extracted directory.
+ncn-m001# rm sat-2.2.x.tar.gz
+ncn-m001# rm -rf sat-2.2.x/
+Upgrade only: Ensure that the environment variable SAT_TAG is not set
+in the ~/.bashrc file on any of the management NCNs.
NOTE: This step should only be required when updating from +Shasta 1.4.1 or Shasta 1.4.2.
+The following example assumes three manager NCNs: ncn-m001, ncn-m002, and ncn-m003,
+and shows output from a system in which no further action is needed.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m003: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+The following example shows that SAT_TAG is set in ~/.bashrc on ncn-m002.
+Remove that line from the ~/.bashrc file on ncn-m002.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+ncn-m002: export SAT_TAG=3.5.0
+ncn-m003: source <(kubectl completion bash)
+Stop the typescript.
+NOTE: This step can be skipped if you wish to use the same typescript +for the remainder of the SAT install. See Next Steps.
+ncn-m001# exit
+SAT version 2.2.x is now installed/upgraded, meaning the SAT 2.2.x release
+has been loaded into the system software repository.
sat command won’t be available until the NCN Personalization
+procedure has been executed.If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the sections listed below.
+NOTE: The NCN Personalization procedure is required when +upgrading SAT. The setup procedures in SAT Setup, however, are +not required when upgrading SAT. They should have been executed +during the first installation of SAT.
+Execute the NCN Personalization procedure:
+ +If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the upgrade procedures:
+ +Describes how to perform NCN personalization using CFS. This personalization process +will configure the System Admin Toolkit (SAT) product stream.
+/tmp/sat-ncn-cfs-configurations.txt./tmp/sat-layer-name.txt./tmp/sat-layer.json....) in shell output indicate omitted lines.2.2.x with the version of the SAT product stream
+being installed.Start a typescript if not already using one.
+The typescript will capture the commands and the output from this installation procedure.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Fresh install only: Add the SAT layer to the NCN personalization JSON file.
+If the SAT install script, install.sh, did not identify and modify the CFS
+configurations that apply to each master management NCN, it will have printed
+the SAT CFS configuration layer in JSON format. This layer must be added to
+the JSON file being used to construct the CFS configuration. For example,
+if the file being used is named ncn-personalization.json, and the SAT
+layer was saved to the file /tmp/sat-layer.json as described in the
+install instructions, the following jq command will append the SAT layer
+and save the result in a new file named ncn-personalization.json.
ncn-m001# jq -s '{layers: (.[0].layers + [.[1]])}' ncn-personalization.json \
+ /tmp/sat-layer.json > ncn-personalization.new.json
+For instructions on how to create a CFS configuration from the previous +file and how to apply it to the management NCNs, refer to “Perform NCN +Personalization” in the HPE Cray System Management Documentation. After +the CFS configuration has been created and applied, return to this +procedure.
+Upgrade only: Invoke each CFS configuration that was updated during the +upgrade.
+If the SAT install script, install.sh, identified CFS configurations that
+apply to the master management NCNs and modified them in place, invoke each
+CFS configuration that was created or updated during installation.
This step will create a CFS session for each given configuration and install +SAT on the associated manager NCNs.
+The --configuration-limit option limits the configuration session to run
+only the SAT layer of the configuration.
You should see a representation of the CFS session in the output.
+ncn-m001# for cfs_configuration in $(cat /tmp/sat-ncn-cfs-configurations.txt);
+do cray cfs sessions create --name "sat-session-${cfs_configuration}" --configuration-name \
+ "${cfs_configuration}" --configuration-limit $(cat /tmp/sat-layer-name.txt);
+done
+
+name="sat-session-ncn-personalization"
+
+[ansible]
+...
+Upgrade only: Monitor the progress of each CFS session.
+This step assumes a single session named sat-session-ncn-personalization was created in the previous step.
First, list all containers associated with the CFS session:
+ncn-m001# kubectl get pod -n services --selector=cfsession=sat-session-ncn-personalization \
+ -o json | jq '.items[0].spec.containers[] | .name'
+"inventory"
+"ansible-1"
+"istio-proxy"
+Next, get the logs for the ansible-1 container.
NOTE: the trailing digit might differ from “1”. It is the zero-based
+index of the sat-ncn layer within the configuration’s layers.
ncn-m001# kubectl logs -c ansible-1 --tail 100 -f -n services \
+ --selector=cfsession=sat-session-ncn-personalization
+Ansible plays, which are run by the CFS session, will install SAT on all the +manager NCNs on the system. Successful results for all of the manager NCN xnames +can be found at the end of the container log. For example:
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Execute this step for each unique CFS configuration.
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify that SAT was successfully configured.
+If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
NOTE: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
ncn-m001# sat --version
+sat 3.7.0
+NOTE: Upon first running sat, you may see additional output while the sat
+container image is downloaded. This will occur the first time sat is run on
+each manager NCN. For example, if you run sat for the first time on ncn-m001
+and then for the first time on ncn-m002, you will see this additional output
+both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+Stop the typescript.
+ncn-m001# exit
+SAT version 2.2.x is now configured:
If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the remaining SAT Setup or SAT Post-Upgrade procedures.
+If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the SAT Post-Upgrade procedures:
+ +Initially, as part of the installation and configuration, SAT authentication is set up so sat commands can be used in
+later steps of the install process. The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For instructions on editing Role Mappings see
+Create Internal User Accounts in the Keycloak Shasta Realm in the CSM product documentation.
+For additional information on SAT authentication, see System Security and Authentication in the CSM
+documentation.
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Some SAT subcommands make requests to the Shasta services through the API gateway and thus require authentication to
+the API gateway in order to function. Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured (see: Generate SAT S3 Credentials). In order to use the SAT S3 bucket,
+the System Administrator must generate the S3 access key and secret keys and write them to a local file. This must be
+done on every Kubernetes manager node where SAT commands are run.
Below is a table describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bmccreds |
+Requires authentication to the API gateway. | +sat-bmccreds |
+Set BMC passwords. | +
sat bootprep |
+Requires authentication to the API gateway. Requires kubernetes configuration and authentication, which is done on ncn-m001 during the install. | +sat-bootprep |
+Prepare to boot nodes with images and configurations. | +
sat bootsys |
+Requires authentication to the API gateway. Requires kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwhist |
+Requires authentication to the API gateway. | +sat-hwhist |
+Report hardware component history. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat k8s |
+Requires kubernetes configuration and authentication, which is automatically configured on ncn-w001 during the install. | +sat-k8s |
+Report on kubernetes replicasets that have co-located replicas (i.e. replicas on the same node). | +
sat linkhealth |
++ | + | This command has been deprecated. | +
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node xnames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat slscheck |
+Requires authentication to the API gateway. | +sat-slscheck |
+Perform a cross-check between SLS and HSM. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC xnames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, you must run the sat auth command. This command will prompt for a password
+on the command line. The username value is obtained from the following locations, in order of higher precedence to lower
+precedence:
--username global command-line option.username option in the api_gateway section of the config file at ~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file will be obtained and saved to
+~/.config/sat/tokens. Subsequent sat commands will determine the username the same way as sat auth described above,
+and will use the token for that username if it has been obtained and saved by sat auth.
sat CLI has been installed following Install The System Admin Toolkit Product Stream.The following is the procedure to globally configure the username used by SAT and authenticate to the API gateway:
+Generate a default SAT configuration file, if one does not exist.
+ncn-m001# sat init
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the config file already exists, it will print out an error:
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway section of the config file. E.g.:
username = "crayadmin"
+Run sat auth. Enter your password when prompted. E.g.:
ncn-m001# sat auth
+Password for crayadmin:
+Succeeded!
+Other sat commands are now authenticated to make requests to the API gateway. E.g.:
ncn-m001# sat status
+Generate S3 credentials and write them to a local file so the SAT user can access S3 storage. In order to use the SAT +S3 bucket, the System Administrator must generate the S3 access key and secret keys and write them to a local file. +This must be done on every Kubernetes master node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the site-specific information set with sat setrev
+(see: Run Sat Setrev to Set System Information).
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Ensure the files are readable only by root.
ncn-m001# touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+ncn-m001# chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+Get the SAT configuration file’s endpoint value.
+NOTE: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
ncn-m001# grep endpoint ~/.config/sat/sat.toml
+# endpoint = "https://rgw-vip.nmn"
+Get the sat-s3-credentials secret’s endpoint value.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, change the SAT configuration file’s endpoint value to match the secret’s.
+Copy SAT configurations to each manager node on the system.
+ncn-m001# for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+NOTE: Depending on how many manager nodes are on the system, the list of manager nodes may +be different. This example assumes three manager nodes, where the configuration files must be +copied from ncn-m001 to ncn-m002 and ncn-m003. Therefore, the list of hosts above is ncn-m002 +and ncn-m003.
+NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Run sat setrev to set System Revision Information. Follow the on-screen prompts to set
+the following site-specific values:
TIP: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. I.e., “System type” is EX-1C.
+ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Run sat showrev to verify System Revision Information. The following tables contain example information.
ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
+In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accomodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
ncn-m001:~ # grep -A 3 logging ~/.config/sat/sat.toml
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checks
+sat sensors
+The following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xname
+sat xname2nid
+sat swap
+prodmgr. Older versions must be uninstalled manually.prodmgr command is available.Use sat showrev to list versions of SAT.
NOTE: It is not recommended to uninstall a version designated as “active”. +If the active version is uninstalled, then the activate procedure must be executed +on a remaining version.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+-------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+-------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+-------------------+-----------------------+
+Use prodmgr to uninstall a version of SAT.
This command will do three things:
+cray-product-catalog Kubernetes ConfigMap, so that it will no longer show up
+in the output of sat showrev.ncn-m001# prodmgr uninstall sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 has been removed.
+Removed Docker image cray/cray-sat:3.9.0
+Removed Docker image cray/sat-cfs-install:1.0.2
+Removed Docker image cray/sat-install-utility:1.4.0
+Deleted sat-2.2.10 from product catalog.
+This procedure can be used to downgrade the active version of SAT.
+prodmgr command is available.Use sat showrev to list versions of SAT.
ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Use prodmgr to activate a different version of SAT.
This command will do three things:
+2.2.10
+sets the repository sat-2.2.10-sle-15sp2 as the only member of the sat-sle-15sp2 group.2.2.10 as active within the product catalog, so that it appears active in the output of
+sat showrev.ncn-personalization). Specifically, it will ensure that the layer refers to the version of SAT CFS
+configuration content associated with the version of SAT being activated.ncn-m001# prodmgr activate sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 is now the default in sat-sle-15sp2.
+Set sat-2.2.10 as active in product catalog.
+Updated CFS configurations: [ncn-personalization]
+Verify that the chosen version is marked as active.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | False | - | - |
+| sat | 2.2.10 | True | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Run NCN Personalization.
+At this point, the command has modified Nexus package repositories to set a particular package repository
+as active, but no packages on the NCNs have been changed. In order to complete the activation process,
+NCN Personalization must be executed to change the cray-sat-podman package version on the manager NCNs.
NOTE: Refer to the command output from step 2 for the names of all CFS configurations that were updated,
+which may not necessarily be just ncn-personalization. If multiple configurations were updated in step 2, then
+a cray cfs sessions create command should be run for each of them. This example assumes a single configuration
+named ncn-personalization was updated. If multiple were updated, set cfs_configurations to a space-separated
+list below.
ncn-m001# cfs_configurations="ncn-personalization"
+ncn-m001# for cfs_configuration in ${cfs_configurations}
+do cray cfs sessions create --name "sat-session-${cfs_configuration}" --configuration-name \
+ "${cfs_configuration}" --configuration-limit sat-ncn;
+done
+Monitor the progress of each CFS session.
+This step assumes a single session named sat-session-ncn-personalization was created in the previous step.
First, list all containers associated with the CFS session:
+ncn-m001# kubectl get pod -n services --selector=cfsession=sat-session-ncn-personalization \
+ -o json | jq '.items[0].spec.containers[] | .name'
+"inventory"
+"ansible-1"
+"istio-proxy"
+Next, get the logs for the ansible-1 container.
NOTE: the trailing digit might differ from “1”. It is the zero-based
+index of the sat-ncn layer within the configuration’s layers.
ncn-m001# kubectl logs -c ansible-1 --tail 100 -f -n services \
+ --selector=cfsession=sat-session-ncn-personalization
+Ansible plays, which are run by the CFS session, will install SAT on all the +manager NCNs on the system. Successful results for all of the manager NCN xnames +can be found at the end of the container log. For example:
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Execute this step for each unique CFS configuration.
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify the new version of the SAT CLI.
+NOTE: This version number will differ from the version number of the SAT +release distribution. This is the semantic version of the SAT Python package, +which is different from the version number of the overall SAT release distribution.
+ncn-m001# sat --version
+3.9.0
+The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands +used on the Cray XC platform. For more information on SAT commands, see System Admin Toolkit Command Overview.
+Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
+Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and +are reported through Redfish.
+SAT is installed as a separate product as part of the HPE Cray EX System base installation.
+Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the +HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are +similarities between SAT commands and xt commands used on the Cray XC platform.
+The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using podman, a daemonless container runtime. SAT runs on Kubernetes
+manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
+sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
+(CONTAINER-ID)sat-container# sat status
+ncn-m001# sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
+example.
ncn-m001# sat-man status
+A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
+ncn-m001# man sat-podman
+The host name in a command prompt indicates where the command must be run. The account that must run the command is +also indicated in the prompt.
+root or super-user account always has the # character at the end of the prompt and has the host name of the
+host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
+referred to as user.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run on one of the Kubernetes Manager servers. (Non-interactive) | +
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
Examples of the sat status command used by an administrator:
ncn-m001# sat status
+ncn-m001# sat bash
+(CONTAINER_ID) sat-container# sat status
+Most sat subcommands depend on services or components from other products in the
+HPE Cray EX (Shasta) software stack. The following list shows these dependencies
+for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
+sat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nidSAT 2.2.16 was released on February 25th, 2022.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container image and Helm chartIt also added the following new components:
+sat-install-utility container imagecfs-config-util container imageThe following sections detail the changes in this release.
+sat command unavailable in sat bash shellAfter launching a shell within the SAT container with sat bash, the sat command will not
+be found. For example:
(CONTAINER-ID) sat-container:~ # sat status
+bash: sat: command not found
+This can be resolved temporarily in one of two ways. /sat/venv/bin/ may be prepended to the
+$PATH environment variable:
(CONTAINER-ID) sat-container:~ # export PATH=/sat/venv/bin:$PATH
+(CONTAINER-ID) sat-container:~ # sat status
+Or, the file /sat/venv/bin/activate may be sourced:
(CONTAINER-ID) sat-container:~ # source /sat/venv/bin/activate
+(CONTAINER-ID) sat-container:~ # sat status
+sat bash shellAfter launching a shell within the SAT container with sat bash, tab completion for sat
+commands does not work.
This can be resolved temporarily by sourcing the file /etc/bash_completion.d/sat-completion.bash:
source /etc/bash_completion.d/sat-completion.bash
+sat in root directorysat commands will not work if the current directory is /. For example:
ncn-m001:/ # sat --help
+Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: open /dev/console: operation not permitted: OCI runtime permission denied error
+To resolve, run sat in another directory.
sat in config directorysat commands will not work if the current directory is ~/.config/sat. For example:
ncn-m001:~/.config/sat # sat --help
+Error: /root/.config/sat: duplicate mount destination
+To resolve, run sat in another directory.
sat commandssat bootprep automates the creation of CFS configurations, the build and
+customization of IMS images, and the creation of BOS session templates. See
+SAT Bootprep for details.sat slscheck performs a check for consistency between the System Layout
+Service (SLS) and the Hardware State Manager (HSM).sat bmccreds provides a simple interface for interacting with the System
+Configuration Service (SCSD) to set BMC Redfish credentials.sat hwhist displays hardware component history by xname (location) or by
+its Field-Replaceable Unit ID (FRUID). This command queries the Hardware
+State Manager (HSM) API to obtain this information. Since the sat hwhist
+command supports querying for the history of a component by its FRUID, the
+FRUID of components has been added to the output of sat hwinv.The following automation has been added to the install script, install.sh:
sat-config-import Kubernetes job, which is
+started when the sat-cfs-install Helm chart is deployed.The SAT product uploads additional information to the cray-product-catalog
+Kubernetes ConfigMap detailing the components it provides, including container
+(Docker) images, Helm charts, RPMs, and package repositories.
This information is used to support uninstall and activation of SAT product +versions moving forward.
+Beginning with the 2.2 release, SAT now provides partial support for the +uninstall and activation of the SAT product stream.
+See Uninstall: Removing a Version of SAT +and Activate: Switching Between Versions +for details.
+sat statusA Subrole column has been added to the output of sat status. This allows you
+to easily differentiate between master, worker, and storage nodes in the
+management role, for example.
Hostname information from SLS has been added to sat status output.
Support for JSON-formatted output has been added to commands which currently
+support the --format option, such as hwinv, status, and showrev.
Many usability improvements have been made to multiple sat commands,
+mostly related to filtering command output. The following are some highlights:
--fields option to display only specific fields for subcommands which
+display tabular reports.--filter queries
+so that the first match is used, similar to --sort-by.--filter, --fields, and --reverse for summaries
+displayed by sat hwinv.sat hwinv.The default log level for stderr has been changed from “WARNING” to “INFO”. For +details, see SAT Logging.
+With the command-line options --loglevel-stderr and --loglevel-file, the log level
+can now be configured separately for stderr and the log file.
The existing --loglevel option is now an alias for the --loglevel-stderr option.
The Podman wrapper script is the script installed at /usr/bin/sat on the
+master management NCNs by the cray-sat-podman RPM that runs the cray-sat
+container in podman. The following subsections detail improvements that were
+made to the wrapper script in this release.
The Podman wrapper script that launches the cray-sat container with podman
+has been modified to mount the user’s current directory and home directory into
+the cray-sat container to provide access to local files in the container.
The man page for the Podman wrapper script, which is accessed by typing man sat on a master management NCN, has been improved to document the following:
Fixed issues with redirecting stdout and stderr, and piping output to commands,
+such as awk, less, and more.
A new sat option has been added to configure the HTTP timeout length for
+requests to the API gateway. See sat-man sat for details.
sat bootsys ImprovementsMany improvements and fixes have been made to sat bootsys. The following are some
+highlights:
--excluded-ncns option, which can be used to omit NCNs
+from the platform-services and ncn-power stages in case they are
+inaccessible.sat bootsys shutdown now prompt the user to
+continue before proceeding. A new option, --disruptive, will bypass this.platform-services
+stage of sat bootsys boot.sat xname2nid Improvementssat xname2nid can now recursively expand slot, chassis, and cabinet xnames to
+a list of nids in those locations.
A new --format option has been added to sat xname2nid. It sets the output format to
+either “range” (the default) or “nid”. The “range” format displays nids in a
+compressed range format suitable for use with a workload manager like Slurm.
The commands which interact with HSM (e.g., sat status and sat hwinv) now
+use the v2 HSM API.
sat diag Limited to HSN Switchessat diag will now only operate against HSN switches by default. These are the
+only controllers that support running diagnostics with HMJTD.
sat showrev EnhancementsA column has been added to the output of sat showrev that indicates whether a
+product version is “active”. The definition of “active” varies across products,
+and not all products may set an “active” version.
For SAT, the active version is the one with its hosted-type package repository in
+Nexus set as the member of the group-type package repository in Nexus,
+meaning that it will be used when installing the cray-sat-podman RPM.
cray-sat Container Image Size ReductionThe size of the cray-sat container image has been approximately cut in half by
+leveraging multi-stage builds. This also improved the repeatability of the unit
+tests by running them in the container.
Minor bug fixes were made in cray-sat and in cray-sat-podman. For full change lists,
+see each repository’s CHANGELOG.md file.
We released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named “ncn-personalization”.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+executables on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and config file options +have been removed. For further instructions, see Remove Obsolete Configuration +File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes masters and workers.
sat firmware.cray-sat container image.sat firmware command.We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes management cluster (i.e., workers
+and masters). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Config File Location ChangeThe default location of the SAT config file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own config files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment variable.
Additionally, if a config file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies option.--list-node-accels option. The count of
+node accelerators is also included for each node.--list-node-accel-risers
+option. The count of node accelerator risers is also included for each node.--list-node-hsn-nics
+option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For detailed +instructions, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding config-file
+option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding config file options were
+deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchSee the System Admin Toolkit Command Overview +and the table of commands in the SAT Authentication section +of this document for more details on each of these commands.
+ + + + + +SAT provides an automated solution for creating CFS configurations, building +and configuring images in IMS, and creating BOS session templates based on a +given input file which defines how those configurations, images, and session +templates should be created.
+This automated process centers around the sat bootprep command. Man page
+documentation for sat bootprep can be viewed similarly to other SAT commands.
ncn-m001# sat-man sat-bootprep
+sat bootprep is used to create CFS configurations, build and
+rename IMS images, and create BOS session templates which tie the
+configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
+including (but not limited to) performing BOS operations (such as creating BOS
+sessions), powering on and off cabinets, and checking the state of the system
+prior to shutdown.
The input file provided to sat bootprep is a YAML-formatted file containing
+information which CFS, IMS, and BOS use to create configurations, images, and
+BOS session templates respectively. Writing and modifying these input files is
+the main task associated with using sat bootprep. An input file is composed of
+three main sections, one each for configurations, images, and session templates.
+These sections may be specified in any order, and any of the sections may be
+omitted if desired.
The configurations section begins with a configurations: key.
---
+configurations:
+Under this key, the user can list one or more configurations to create. For +each configuration, a name should be given, in addition to the list of layers +which comprise the configuration. Each layer can be defined by a product name +and optionally a version number, or commit hash or branch in the product’s +configuration repository. Alternatively, a layer can be defined by a Git +repository URL directly, along with an associated branch or commit hash.
+When a configuration layer is specified in terms of a product name, the layer +is created in CFS by looking up relevant configuration information (including +the configuration repository and commit information) from the +cray-product-catalog Kubernetes ConfigMap as necessary. A version may be +supplied, but if it is absent, the version is assumed to be the latest version +found in the cray-product-catalog.
+---
+configurations:
+- name: example-configuration
+ layers:
+ - name: example product
+ playbook: example.yml
+ product:
+ name: example
+ version: 1.2.3
+Alternatively, a configuration layer may be specified by explicitly referencing
+the desired configuration repository, along with the branch containing the
+intended version of the Ansible playbooks. A commit hash may be specified by replacing
+branch with commit.
...
+ - name: another example product
+ playbook: another-example.yml
+ git:
+ url: "https://vcs.local/vcs/another-example-config-management.git"
+ branch: main
+ ...
+When sat bootprep is run against an input file, a CFS configuration will be
+created corresponding to each configuration in the configurations section. For
+example, the configuration created from an input file with the layers listed
+above might look something like the following:
{
+ "lastUpdated": "2022-02-07T21:47:49Z",
+ "layers": [
+ {
+ "cloneUrl": "https://vcs.local/vcs/example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "example product",
+ "playbook": "example.yml"
+ },
+ {
+ "cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "another example product",
+ "playbook": "another-example.yml"
+ }
+ ],
+ "name": "example-configuration"
+}
+After specifying configurations, the user may add images to the input file
+which are to be built by IMS. To add an images section, the user should add
+an images key.
---
+configurations:
+ ... (omitted for brevity)
+images:
+Under the images key, the user may define one or more images to be created in
+a list. Each element of the list defines a separate IMS image to be built and/or
+configured. Images must contain a name, as well as an ims section containing a
+definition of the image to be built and/or configured. Images may be defined by
+an image recipe, or by a pre-built image. Recipes and pre-built images are
+referred to by their names or IDs in IMS. The ims section should also contain
+an is_recipe property, which indicates whether the name or ID refers to an
+image recipe or a pre-built image. Images may also optionally provide a text
+description of the image. This description is not stored or used by sat bootprep or any CSM services, but is useful for documenting images in the input
+file.
---
+configurations:
+ ... (omitted for brevity)
+images:
+- name: example-compute-image
+ description: >
+ An example compute node image for illustrative purposes.
+ ims:
+ name: example-compute-image-recipe
+ is_recipe: true
+- name: another-example-compute-image
+ description: >
+ Another example compute node image.
+ ims:
+ id: <IMS image UUID>
+ is_recipe: false
+Images may also contain a configuration property in their definition, which
+specifies a configuration with which to customize the built image prior to
+booting. If a configuration is specified, then configuration groups must also
+be specified using the configuration_group_names property.
---
+configurations:
+ ... (omitted for brevity)
+images:
+- name: example-compute-image
+ description: >
+ An example compute node image for illustrative purposes.
+ ims:
+ name: example-compute-image-recipe
+ is_recipe: true
+ configuration: example configuration
+ configuration_group_names:
+ - Compute
+BOS session templates are the final section of the input file, and are defined
+under the session_templates key.
---
+configurations:
+ ... (omitted for brevity)
+images:
+ ... (omitted for brevity)
+session_templates:
+Each session template is defined in terms of its name, an image, a
+configuration, and a set of parameters which can be used to configure the
+session. The name, image, and configuration are specified with their respective
+name, image, and configuration keys. bos_parameters may also be
+specified; currently, the only setting under bos_parameters that is supported
+is boot_sets, which can be used to define boot sets in the BOS session
+template. Each boot set is defined under its own property under boot_sets, and
+the value of each boot set can contain the following properties, all of
+which are optional:
kernel_parameters: the parameters passed to the kernel on the command linenetwork: the network over which the nodes will bootnode_list: nodes to add to the boot setnode_roles_groups: HSM roles to add to the boot setnode_groups: HSM groups to add to the boot setrootfs_provider: the root file system providerrootfs_provider_passthrough: parameters to add to the rootfs= kernel
+parameterThe properties listed previously are the same as the parameters that can be +specified directly through BOS boot sets. More information can be found in the +CSM documentation on session +templates. +Additional properties not listed are passed through to the BOS session template +as written.
+An example session template might look like the following:
+configurations:
+ ... (omitted for brevity)
+images:
+ ... (omitted for brevity)
+session_templates:
+- name: example-session-template
+ image: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_list: []
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+Putting together all of the previous input file sections, an example bootprep input +file might look something like the following.
+---
+configurations:
+- name: cos-config
+ layers:
+ - name: cos-integration-2.2.87
+ playbook: site.yml
+ product:
+ name: cos
+ version: 2.2.87
+ branch: integration
+ - name: cpe-integration-21.12.3
+ playbook: pe_deploy.yml
+ product:
+ name: cpe
+ version: 21.12.3
+ branch: integration
+ - name: slurm-master-1.1.1
+ playbook: site.yml
+ product:
+ name: slurm
+ version: 1.1.1
+ branch: master
+images:
+- name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ ims:
+ is_recipe: true
+ name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ configuration: cos-config
+ configuration_group_names:
+ - Compute
+session_templates:
+- name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ image: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ configuration: cos-config
+ bos_parameters:
+ boot_sets:
+ compute:
+ kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
+ node_roles_groups:
+ - Compute
+It is possible to create an example bootprep input file using values from the
+system’s product catalog using the sat bootprep generate-example command.
ncn-m001# sat bootprep generate-example
+INFO: Using latest version (2.3.24-20220113160653) of product cos
+INFO: Using latest version (21.11.4) of product cpe
+INFO: Using latest version (1.0.7) of product slurm
+INFO: Using latest version (1.1.24) of product analytics
+INFO: Using latest version (2.1.5) of product uan
+INFO: Using latest version (21.11.4) of product cpe
+INFO: Using latest version (1.0.7) of product slurm
+INFO: Using latest version (1.1.24) of product analytics
+INFO: Using latest version (2.3.24-20220113160653) of product cos
+INFO: Using latest version (2.1.5) of product uan
+INFO: Wrote example bootprep input file to ./example-bootprep-input.yaml.
+This file should be reviewed and edited to match the desired parameters of the +configurations, images, and session templates.
+The contents of the YAML input files described above must conform to a schema +which defines the structure of the data. The schema definition is written using +the JSON Schema format. (Although the format is named “JSON Schema”, the schema +itself is written in YAML as well.) More information, including introductory +materials and a formal specification of the JSON Schema metaschema, can be found +on the JSON Schema website.
+To view the exact schema specification, run sat bootprep view-schema.
ncn-m001# sat bootprep view-schema
+---
+$schema: "https://json-schema.org/draft-07/schema"
+title: Bootprep Input File
+description: >
+ A description of the set of CFS configurations to create, the set of IMS
+ images to create and optionally customize with the defined CFS configurations,
+ and the set of BOS session templates to create that reference the defined
+ images and configurations.
+type: object
+additionalProperties: false
+properties:
+ ...
+The raw schema definition can be difficult to understand without experience +working with JSON Schema specifications. For this reason, a feature was included +which can generate user-friendly HTML documentation for the input file schema +which can be browsed with the user’s preferred web browser.
+Create a documentation tarball using sat bootprep.
ncn-m001# sat bootprep generate-docs
+INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
+An alternate output directory can be specified with the --output-dir
+option. The generated tarball is always named bootprep-schema-docs.tar.gz.
ncn-m001# sat bootprep generate-docs --output-dir /tmp
+INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
+From another machine, copy the tarball to a local directory.
+another-machine$ scp root@ncn-m001:bootprep-schema-docs.tar.gz .
+Extract the contents of the tarball and open the contained index.html.
another-machine$ tar xzvf bootprep-schema-docs.tar.gz
+x bootprep-schema-docs/
+x bootprep-schema-docs/index.html
+x bootprep-schema-docs/schema_doc.css
+x bootprep-schema-docs/schema_doc.min.js
+another-machine$ open bootprep-schema-docs/index.html
+The SAT Grafana Dashboards display messages that are generated by the HSN (High Speed Network) and reported through +Redfish. The messages are displayed based on severity.
+Grafana can be accessed via web browser at the following URL:
+https://sma-grafana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Grafana being https://sma-grafana.cmn.EXAMPLE_DOMAIN.com
For additional details about how to access the Grafana Dashboards refer to Access the Grafana Monitoring UI in the +SMA product documentation.
+For more information about the interpretation of metrics for the SAT Grafana Dashboards refer to Fabric Telemetry +Kafka Topics in the SMA product documentation.
+There are four Fabric Telemetry dashboards used in SAT that report on the HSN. Two contain chart panels and two display +telemetry in a tabular format.
+| Dashboard Name | +Display Type | +
|---|---|
| Fabric Congestion | +Chart Panels | +
| Fabric RFC3635 | +Chart Panels | +
| Fabric Errors | +Tabular Format | +
| Fabric Port State | +Tabular Format | +
The tabular format presents a single point of telemetry for a given location and metric, either because the telemetry +is not numerical or that it changes infrequently. The value shown is the most recently reported value for that location +during the time range selected, if any. The interval setting is not used for tabular dashboards.
+Shows the Interval and Locations Options for the available telemetry.
+
The value of the Interval option sets the time resolution of the received telemetry. This works a bit like a +histogram, with the available telemetry in an interval of time going into a “bucket” and averaging out to a single +point on the chart or table. The special value auto will choose an interval based on the time range selected.
+For additional information, refer to Grafana Templates and Variables.
+The Locations option allows restriction of the telemetry shown by locations, either individual links or all links +in a switch. The selection presented updates dynamically according to time range, except for the errors dashboard, +which always has entries for all links and switches, although the errors shown are restricted to the selected time +range.
+The chart panels for the RFC3635 and Congestion dashboards allow selection of a single location from the chart’s legend +or the trace on the chart.
+
SAT Grafana Dashboards provide system administrators a way to view fabric telemetry data across all Rosetta switches in +the system and assess the past and present health of the high-speed network. It also allows the ability to drill down +to view data for specific ports on specific switches.
+This dashboard contains the variable, Port Type not found in the other dashboards. The possible values are edge, +local, and global and correspond to the link’s relationship to the network topology. The locations presented in the +panels are restricted to the values (any combination, defaults to “all”) selected.
+The metric values for links of a given port type are similar in value to each other but very distinct from the values of +other types. If the values for different port types are all plotted together, the values for links with lower values are +indistinguishable from zero when plotted.
+The port type of a link is reported as a port state “subtype” event when defined at port initialization.
+
This dashboard reports error counters in a tabular format in three panels.
+There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+Unlike other dashboards, the locations presented are all locations in the system rather than having telemetry within +the time range selected. However, the values are taken from telemetry within the time range.
+
There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+The Fabric Port State telemetry is distinct because it typically is not numeric. It also updates infrequently, so a +long time range may be necessary to obtain any values. Port State is refreshed daily, so a time range of 24 hours +results in all states for all links in the system being shown.
+The three columns named, group, switch, and port are not port state events, but extra information included with +all port state events.
+
For additional information on performance counters, refer to +Definitions of Managed Objects for the Ethernet-like Interface Types, +an Internet standards document.
+Because these metrics are counters that only increase over time, the values plotted are the change in the counter’s +value over the interval setting.
+ + + + + +Kibana is an open source analytics and visualization platform designed to search, view, and interact with data stored +in Elasticsearch indices. Kibana runs as a web service and has a browser-based interface. It offers visual output of +node data in the forms of charts, tables and maps that display real-time Elasticsearch queries. Viewing system data in +this way breaks down the complexity of large data volumes into easily understood information.
+Kibana can be accessed via web browser at the following URL:
+https://sma-kibana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Kibana being https://sma-kibana.cmn.EXAMPLE_DOMAIN.com
For additional details about how to access the Kibana Dashboards refer to View Logs Via Kibana in the SMA product +documentation.
+Additional details about the AER, ATOM, Heartbeat, Kernel, MCE, and Rasdaemon Kibana Dashboards are included in this +table.
+| Dashboard | +Short Description | +Long Description | +Kibana Visualization and Search Name | +
|---|---|---|---|
| sat-aer | +AER corrected | +Corrected Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-corrected Search: sat-aer-corrected | +
| sat-aer | +AER fatal | +Fatal Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-fatal Search: sat-aer-fatal | +
| sat-atom | +ATOM failures | +Application Task Orchestration and Management tests are run on a node when a job finishes. Test failures are logged. | +sat-atom-failed | +
| sat-atom | +ATOM admindown | +Application Task Orchestration and Management test failures can result in nodes being marked admindown. An admindown node is not available for job launch. | +sat-atom-admindown | +
| sat-heartbeat | +Heartbeat loss events | +Heartbeat loss event messages reported by the hbtd pods that monitor for heartbeats across nodes in the system. | +sat-heartbeat | +
| sat-kernel | +Kernel assertions | +The kernel software performs a failed assertion when some condition represents a serious fault. The node goes down. | +sat-kassertions | +
| sat-kernel | +Kernel panics | +The kernel panics when something is seriously wrong. The node goes down. | +sat-kernel-panic | +
| sat-kernel | +Lustre bugs (LBUGs) | +The Lustre software in the kernel stack performs a failed assertion when some condition related to file system logic represents a serious fault. The node goes down. | +sat-lbug | +
| sat-kernel | +CPU stalls | +CPU stalls are serous conditions that can reduce node performance, and sometimes cause a node to go down. Technically these are Read-Copy-Update stalls where software in the kernel stack holds onto memory for too long. Read-Copy-Update is a vital aspect of kernel performance and rather esoteric. | +sat-cpu-stall | +
| sat-kernel | +Out of memory | +An Out Of Memory (OOM) condition has occurred. The kernel must kill a process to continue. The kernel will select an expendable process when possible. If there is no expendable process the node usually goes down in some manner. Even if there are expendable processes the job is likely to be impacted. OOM conditions are best avoided. | +sat-oom | +
| sat-mce | +MCE | +Machine Check Exceptions (MCE) are errors detected at the processor level. | +sat-mce | +
| sat-rasdaemon | +rasdaemon errors | +Errors from the rasdaemon service on nodes. The rasdaemon service is the Reliability, Availability, and Serviceability Daemon, and it is intended to collect all hardware error events reported by the linux kernel, including PCI and MCE errors. This may include certain HSN errors in the future. |
+sat-rasdaemon-error | +
| sat-rasdaemon | +rasdaemon messages | +All messages from the rasdaemon service on nodes. |
+sat-rasdaemon | +
By default, search highlighting is enabled. This procedure instructs how to disable search highlighting.
+The Kibana Dashboard should be open on your system.
+Navigate to Management
+Navigate to Advanced Settings in the Kibana section, below the Elastic search section
+Scroll down to the Discover section
+Change Highlight results from on to off
+Click Save to save changes
+The AER Dashboard displays errors that come from the PCI Express Advanced Error Reporting (AER) driver. These errors +are split up into separate visualizations depending on whether they are fatal or corrected errors.
+Go to the dashboard section.
+Select sat-aer dashboard.
+Choose the time range of interest.
+View the Corrected and Fatal Advanced Error Reporting messages from PCI Express devices on each node. View the +matching log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on +the left. If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass +next to each NID.
+The ATOM (Application Task Orchestration and Management) Dashboard displays node failures that occur during health +checks and application test failures. Some test failures are of possible interest even though a node is not marked +admindown or otherwise fails. They are of clear interest if a node is marked admindown, and might provide +clues if a node otherwise fails. They might also show application problems.
+HPE Cray EX is installed on the system along with the System Admin Toolkit, which contains the ATOM Kibana Dashboard.
+Go to the dashboard section.
+Select sat-atom dashboard.
+Choose the time range of interest.
+View any nodes marked admindown and any ATOM test failures. These failures occur during health checks and +application test failures. Test failures marked admindown are important to note. View the matching log messages +in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. If desired, +results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Heartbeat Dashboard displays heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods +are responsible for monitoring nodes in the system for heartbeat loss.
+Go to the dashboard section.
+Select sat-heartbeat dashboard.
+Choose the time range of interest.
+View the heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods are responsible for +monitoring nodes in the system for heartbeat loss.View the matching log messages in the panel.
+The Kernel Dashboard displays compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. +The messages reveal if Lustre has experienced a fatal error on any compute nodes in the system. A CPU stall is a serious +problem that might result in a node failure. Out-of-memory conditions can be due to applications or system problems and +may require expert analysis. They provide useful clues for some node failures and may reveal if an application is using +too much memory.
+Go to the dashboard section.
+Select sat-kernel dashboard.
+Choose the time range of interest.
+View the compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. View the matching +log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. +If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to +each NID.
+The MCE Dashboard displays CPU detected processor-level hardware errors.
+Go to the dashboard section.
+Select sat-mce dashboard.
+Choose the time range of interest.
+View the Machine Check Exceptions (MCEs) listed including the counts per NID (node). For an MCE, the CPU number and +DIMM number can be found in the message, if applicable. View the matching log messages in the panel(s) on the right, +and view the counts of each message per NID in the panel(s) on the left. If desired, results can be filtered by NID +by clicking the icon showing a + inside a magnifying glass next to each NID.
+The Rasdaemon Dashboard displays errors that come from the Reliability, Availability, and Serviceability (RAS) daemon
+service on nodes in the system. This service collects all hardware error events reported by the linux kernel, including
+PCI and MCE errors. As a result there may be some duplication between the messages presented here and the messages
+presented in the MCE and AER dashboards. This dashboard splits up the messages into two separate visualizations, one
+for only messages of severity “emerg” or “err” and another for all messages from rasdaemon.
Go to the dashboard section.
+Select sat-rasdaemon dashboard.
+Choose the time range of interest.
+View the errors that come from the Reliability, Availability, and Serviceability (RAS) daemon service on nodes in +the system. View the matching log messages in the panel(s) on the right, and view the counts of each message per NID +in the panel(s) on the left. If desired, results can be filtered by NID by clicking the icon showing a + inside +a magnifying glass next to each NID.
+Describes how to install the System Admin Toolkit (SAT) product stream.
+...) in shell output indicate omitted lines.2.2.x with the version of the SAT product stream
+being installed.Start a typescript.
+The typescript will record the commands and the output from this installation.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Copy the release distribution gzipped tar file to ncn-m001.
Unzip and extract the release distribution, 2.2.x.
ncn-m001# tar -xvzf sat-2.2.x.tar.gz
+Change directory to the extracted release distribution directory.
+ncn-m001# cd sat-2.2.x
+Run the installer: install.sh.
+The script produces a lot of output. A successful install ends with “SAT +version 2.2.x has been installed”.
+ncn-m001# ./install.sh
+...
+====> Updating active CFS configurations
+...
+====> SAT version 2.2.x has been installed.
+Upgrade only: Record the names of the CFS configuration or
+configurations modified by install.sh.
The install.sh script attempts to modify any CFS configurations that apply
+to the master management NCNs. During an upgrade, install.sh will log
+messages indicating the CFS configuration or configurations that were
+modified. For example, if there are three master nodes all using the same
+CFS configuration named “ncn-personalization”, the output would look like
+this:
====> Updating active CFS configurations
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, x3000c0s3b0n0, x3000c0s5b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s1b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s3b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s5b0n0
+INFO: Updating CFS configuration "ncn-personalization"
+INFO: Updating existing layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml in configuration "ncn-personalization".
+INFO: Key "name" in layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml updated from sat-ncn to sat-2.2.16
+INFO: Successfully updated layers in configuration "ncn-personalization"
+Save the name of each CFS configuration updated by the installer. In the +previous example, a single configuration named “ncn-personalization” was +updated, so that name is saved to a temporary file.
+ncn-m001# echo ncn-personalization >> /tmp/sat-ncn-cfs-configurations.txt
+Repeat the previous command for each CFS configuration that was updated.
+Upgrade only: Save the new name of the SAT CFS configuration layer.
+In the example install.sh output above, the new layer name is
+sat-2.2.16. Save this value to a file to be used later.
ncn-m001# echo sat-2.2.16 > /tmp/sat-layer-name.txt
+Fresh install only: Save the CFS configuration layer for SAT to a file +for later use.
+The install.sh script attempts to modify any CFS configurations that apply
+to the master management NCNs. During a fresh install, no such CFS
+configurations will be found, and it will instead log the SAT configuration
+layer that must be added to the CFS configuration that will be created. Here
+is an example of the output in that case:
====> Updating active CFS configurations
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, x3000c0s3b0n0, x3000c0s5b0n0
+WARNING: No CFS configurations found that apply to components with role Management and subrole Master.
+INFO: The following sat layer should be used in the CFS configuration that will be applied to NCNs with role Management and subrole Master.
+{
+ "name": "sat-2.2.15",
+ "commit": "9a74b8f5ba499af6fbcecfd2518a40e081312933",
+ "cloneUrl": "https://api-gw-service-nmn.local/vcs/cray/sat-config-management.git",
+ "playbook": "sat-ncn.yml"
+}
+Save the JSON output to a file for later use. For example:
+ncn-m001# cat > /tmp/sat-layer.json <<EOF
+> {
+> "name": "sat-2.2.15",
+> "commit": "9a74b8f5ba499af6fbcecfd2518a40e081312933",
+> "cloneUrl": "https://api-gw-service-nmn.local/vcs/cray/sat-config-management.git",
+> "playbook": "sat-ncn.yml"
+> }
+> EOF
+Do not copy the previous command verbatim. Use the JSON output from the
+install.sh script.
Optional: Remove the SAT release distribution tar file and extracted directory.
+ncn-m001# rm sat-2.2.x.tar.gz
+ncn-m001# rm -rf sat-2.2.x/
+Upgrade only: Ensure that the environment variable SAT_TAG is not set
+in the ~/.bashrc file on any of the management NCNs.
NOTE: This step should only be required when updating from +Shasta 1.4.1 or Shasta 1.4.2.
+The following example assumes three manager NCNs: ncn-m001, ncn-m002, and ncn-m003,
+and shows output from a system in which no further action is needed.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m003: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+The following example shows that SAT_TAG is set in ~/.bashrc on ncn-m002.
+Remove that line from the ~/.bashrc file on ncn-m002.
ncn-m001# pdsh -w ncn-m00[1-3] cat ~/.bashrc
+ncn-m001: source <(kubectl completion bash)
+ncn-m002: source <(kubectl completion bash)
+ncn-m002: export SAT_TAG=3.5.0
+ncn-m003: source <(kubectl completion bash)
+Stop the typescript.
+NOTE: This step can be skipped if you wish to use the same typescript +for the remainder of the SAT install. See Next Steps.
+ncn-m001# exit
+SAT version 2.2.x is now installed/upgraded, meaning the SAT 2.2.x release
+has been loaded into the system software repository.
sat command won’t be available until the NCN Personalization
+procedure has been executed.If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the sections listed below.
+NOTE: The NCN Personalization procedure is required when +upgrading SAT. The setup procedures in SAT Setup, however, are +not required when upgrading SAT. They should have been executed +during the first installation of SAT.
+Execute the NCN Personalization procedure:
+ +If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the upgrade procedures:
+ +Describes how to perform NCN personalization using CFS. This personalization process +will configure the System Admin Toolkit (SAT) product stream.
+/tmp/sat-ncn-cfs-configurations.txt./tmp/sat-layer-name.txt./tmp/sat-layer.json....) in shell output indicate omitted lines.2.2.x with the version of the SAT product stream
+being installed.Start a typescript if not already using one.
+The typescript will capture the commands and the output from this installation procedure.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Fresh install only: Add the SAT layer to the NCN personalization JSON file.
+If the SAT install script, install.sh, did not identify and modify the CFS
+configurations that apply to each master management NCN, it will have printed
+the SAT CFS configuration layer in JSON format. This layer must be added to
+the JSON file being used to construct the CFS configuration. For example,
+if the file being used is named ncn-personalization.json, and the SAT
+layer was saved to the file /tmp/sat-layer.json as described in the
+install instructions, the following jq command will append the SAT layer
+and save the result in a new file named ncn-personalization.json.
ncn-m001# jq -s '{layers: (.[0].layers + [.[1]])}' ncn-personalization.json \
+ /tmp/sat-layer.json > ncn-personalization.new.json
+For instructions on how to create a CFS configuration from the previous +file and how to apply it to the management NCNs, refer to “Perform NCN +Personalization” in the HPE Cray System Management Documentation. After +the CFS configuration has been created and applied, return to this +procedure.
+Upgrade only: Invoke each CFS configuration that was updated during the +upgrade.
+If the SAT install script, install.sh, identified CFS configurations that
+apply to the master management NCNs and modified them in place, invoke each
+CFS configuration that was created or updated during installation.
This step will create a CFS session for each given configuration and install +SAT on the associated manager NCNs.
+The --configuration-limit option limits the configuration session to run
+only the SAT layer of the configuration.
You should see a representation of the CFS session in the output.
+ncn-m001# for cfs_configuration in $(cat /tmp/sat-ncn-cfs-configurations.txt);
+do cray cfs sessions create --name "sat-session-${cfs_configuration}" --configuration-name \
+ "${cfs_configuration}" --configuration-limit $(cat /tmp/sat-layer-name.txt);
+done
+
+name="sat-session-ncn-personalization"
+
+[ansible]
+...
+Upgrade only: Monitor the progress of each CFS session.
+This step assumes a single session named sat-session-ncn-personalization was created in the previous step.
First, list all containers associated with the CFS session:
+ncn-m001# kubectl get pod -n services --selector=cfsession=sat-session-ncn-personalization \
+ -o json | jq '.items[0].spec.containers[] | .name'
+"inventory"
+"ansible-1"
+"istio-proxy"
+Next, get the logs for the ansible-1 container.
NOTE: the trailing digit might differ from “1”. It is the zero-based
+index of the sat-ncn layer within the configuration’s layers.
ncn-m001# kubectl logs -c ansible-1 --tail 100 -f -n services \
+ --selector=cfsession=sat-session-ncn-personalization
+Ansible plays, which are run by the CFS session, will install SAT on all the +manager NCNs on the system. Successful results for all of the manager NCN xnames +can be found at the end of the container log. For example:
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Execute this step for each unique CFS configuration.
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify that SAT was successfully configured.
+If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
NOTE: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
ncn-m001# sat --version
+sat 3.7.0
+NOTE: Upon first running sat, you may see additional output while the sat
+container image is downloaded. This will occur the first time sat is run on
+each manager NCN. For example, if you run sat for the first time on ncn-m001
+and then for the first time on ncn-m002, you will see this additional output
+both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+Stop the typescript.
+ncn-m001# exit
+SAT version 2.2.x is now configured:
If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +to determine which step to execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the remaining SAT Setup or SAT Post-Upgrade procedures.
+If performing a fresh install, execute the SAT Setup procedures:
+ +If performing an upgrade, execute the SAT Post-Upgrade procedures:
+ +Initially, as part of the installation and configuration, SAT authentication is set up so sat commands can be used in
+later steps of the install process. The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For instructions on editing Role Mappings see
+Create Internal User Accounts in the Keycloak Shasta Realm in the CSM product documentation.
+For additional information on SAT authentication, see System Security and Authentication in the CSM
+documentation.
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Some SAT subcommands make requests to the Shasta services through the API gateway and thus require authentication to
+the API gateway in order to function. Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured (see: Generate SAT S3 Credentials). In order to use the SAT S3 bucket,
+the System Administrator must generate the S3 access key and secret keys and write them to a local file. This must be
+done on every Kubernetes manager node where SAT commands are run.
Below is a table describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bmccreds |
+Requires authentication to the API gateway. | +sat-bmccreds |
+Set BMC passwords. | +
sat bootprep |
+Requires authentication to the API gateway. Requires kubernetes configuration and authentication, which is done on ncn-m001 during the install. | +sat-bootprep |
+Prepare to boot nodes with images and configurations. | +
sat bootsys |
+Requires authentication to the API gateway. Requires kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwhist |
+Requires authentication to the API gateway. | +sat-hwhist |
+Report hardware component history. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat k8s |
+Requires kubernetes configuration and authentication, which is automatically configured on ncn-w001 during the install. | +sat-k8s |
+Report on kubernetes replicasets that have co-located replicas (i.e. replicas on the same node). | +
sat linkhealth |
++ | + | This command has been deprecated. | +
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node xnames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat slscheck |
+Requires authentication to the API gateway. | +sat-slscheck |
+Perform a cross-check between SLS and HSM. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC xnames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, you must run the sat auth command. This command will prompt for a password
+on the command line. The username value is obtained from the following locations, in order of higher precedence to lower
+precedence:
--username global command-line option.username option in the api_gateway section of the config file at ~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file will be obtained and saved to
+~/.config/sat/tokens. Subsequent sat commands will determine the username the same way as sat auth described above,
+and will use the token for that username if it has been obtained and saved by sat auth.
sat CLI has been installed following Install The System Admin Toolkit Product Stream.The following is the procedure to globally configure the username used by SAT and authenticate to the API gateway:
+Generate a default SAT configuration file, if one does not exist.
+ncn-m001# sat init
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the config file already exists, it will print out an error:
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway section of the config file. E.g.:
username = "crayadmin"
+Run sat auth. Enter your password when prompted. E.g.:
ncn-m001# sat auth
+Password for crayadmin:
+Succeeded!
+Other sat commands are now authenticated to make requests to the API gateway. E.g.:
ncn-m001# sat status
+Generate S3 credentials and write them to a local file so the SAT user can access S3 storage. In order to use the SAT +S3 bucket, the System Administrator must generate the S3 access key and secret keys and write them to a local file. +This must be done on every Kubernetes master node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the site-specific information set with sat setrev
+(see: Run Sat Setrev to Set System Information).
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Ensure the files are readable only by root.
ncn-m001# touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+ncn-m001# chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+Get the SAT configuration file’s endpoint value.
+NOTE: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
ncn-m001# grep endpoint ~/.config/sat/sat.toml
+# endpoint = "https://rgw-vip.nmn"
+Get the sat-s3-credentials secret’s endpoint value.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, change the SAT configuration file’s endpoint value to match the secret’s.
+Copy SAT configurations to each manager node on the system.
+ncn-m001# for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+NOTE: Depending on how many manager nodes are on the system, the list of manager nodes may +be different. This example assumes three manager nodes, where the configuration files must be +copied from ncn-m001 to ncn-m002 and ncn-m003. Therefore, the list of hosts above is ncn-m002 +and ncn-m003.
+NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Run sat setrev to set System Revision Information. Follow the on-screen prompts to set
+the following site-specific values:
TIP: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. I.e., “System type” is EX-1C.
+ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Run sat showrev to verify System Revision Information. The following tables contain example information.
ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
+In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accomodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
ncn-m001:~ # grep -A 3 logging ~/.config/sat/sat.toml
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checks
+sat sensors
+The following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xname
+sat xname2nid
+sat swap
+prodmgr.prodmgr command is available.Use sat showrev to list versions of SAT.
NOTE: It is not recommended to uninstall a version designated as “active”. +If the active version is uninstalled, then the activate procedure must be executed +on a remaining version.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+-------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+-------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+-------------------+-----------------------+
+Use prodmgr to uninstall a version of SAT.
This command will do three things:
+cray-product-catalog Kubernetes ConfigMap, so that it will no longer show up
+in the output of sat showrev.ncn-m001# prodmgr uninstall sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 has been removed.
+Removed Docker image cray/cray-sat:3.9.0
+Removed Docker image cray/sat-cfs-install:1.0.2
+Removed Docker image cray/sat-install-utility:1.4.0
+Deleted sat-2.2.10 from product catalog.
+This procedure can be used to downgrade the active version of SAT.
+prodmgr command is available.Use sat showrev to list versions of SAT.
ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Use prodmgr to activate a different version of SAT.
This command will do three things:
+2.2.10
+sets the repository sat-2.2.10-sle-15sp2 as the only member of the sat-sle-15sp2 group.2.2.10 as active within the product catalog, so that it appears active in the output of
+sat showrev.ncn-personalization). Specifically, it will ensure that the layer refers to the version of SAT CFS
+configuration content associated with the version of SAT being activated.ncn-m001# prodmgr activate sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 is now the default in sat-sle-15sp2.
+Set sat-2.2.10 as active in product catalog.
+Updated CFS configurations: [ncn-personalization]
+Verify that the chosen version is marked as active.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | False | - | - |
+| sat | 2.2.10 | True | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Run NCN Personalization.
+At this point, the command has modified Nexus package repositories to set a particular package repository
+as active, but no packages on the NCNs have been changed. In order to complete the activation process,
+NCN Personalization must be executed to change the cray-sat-podman package version on the manager NCNs.
NOTE: Refer to the command output from step 2 for the names of all CFS configurations that were updated,
+which may not necessarily be just ncn-personalization. If multiple configurations were updated in step 2, then
+a cray cfs sessions create command should be run for each of them. This example assumes a single configuration
+named ncn-personalization was updated. If multiple were updated, set cfs_configurations to a space-separated
+list below.
ncn-m001# cfs_configurations="ncn-personalization"
+ncn-m001# for cfs_configuration in ${cfs_configurations}
+do cray cfs sessions create --name "sat-session-${cfs_configuration}" --configuration-name \
+ "${cfs_configuration}" --configuration-limit sat-ncn;
+done
+Monitor the progress of each CFS session.
+This step assumes a single session named sat-session-ncn-personalization was created in the previous step.
First, list all containers associated with the CFS session:
+ncn-m001# kubectl get pod -n services --selector=cfsession=sat-session-ncn-personalization \
+ -o json | jq '.items[0].spec.containers[] | .name'
+"inventory"
+"ansible-1"
+"istio-proxy"
+Next, get the logs for the ansible-1 container.
NOTE: the trailing digit might differ from “1”. It is the zero-based
+index of the sat-ncn layer within the configuration’s layers.
ncn-m001# kubectl logs -c ansible-1 --tail 100 -f -n services \
+ --selector=cfsession=sat-session-ncn-personalization
+Ansible plays, which are run by the CFS session, will install SAT on all the +manager NCNs on the system. Successful results for all of the manager NCN xnames +can be found at the end of the container log. For example:
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Execute this step for each unique CFS configuration.
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify the new version of the SAT CLI.
+NOTE: This version number will differ from the version number of the SAT +release distribution. This is the semantic version of the SAT Python package, +which is different from the version number of the overall SAT release distribution.
+ncn-m001# sat --version
+3.9.0
+The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands +used on the Cray XC platform. For more information on SAT commands, see System Admin Toolkit Command Overview.
+Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
+Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and +are reported through Redfish.
+SAT is installed as a separate product as part of the HPE Cray EX System base installation.
+Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the +HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are +similarities between SAT commands and xt commands used on the Cray XC platform.
+The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using podman, a daemonless container runtime. SAT runs on Kubernetes
+manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
+sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
+(CONTAINER-ID)sat-container# sat status
+ncn-m001# sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
+example.
ncn-m001# sat-man status
+A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
+ncn-m001# man sat-podman
+The host name in a command prompt indicates where the command must be run. The account that must run the command is +also indicated in the prompt.
+root or super-user account always has the # character at the end of the prompt and has the host name of the
+host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
+referred to as user.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run on one of the Kubernetes Manager servers. (Non-interactive) | +
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
Examples of the sat status command used by an administrator:
ncn-m001# sat status
+ncn-m001# sat bash
+(CONTAINER_ID) sat-container# sat status
+Most sat subcommands depend on services or components from other products in the
+HPE Cray EX (Shasta) software stack. The following list shows these dependencies
+for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
+sat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nidThe 2.3.4 version of the SAT product includes:
+sat CommandsNone.
+When running sat commands, the current working directory is now mounted in the
+container as /sat/share, and the current working directory within the container
+is also /sat/share.
Files in the current working directory must be specified using relative paths to
+that directory, because the current working directory is always mounted on /sat/share.
+Absolute paths should be avoided, and paths that are outside of $HOME or $PWD
+are never accessible to the container environment.
The home directory is still mounted on the same path inside the container as it +is on the host.
+sat bootsysThe following options were added to sat bootsys.
--bos-limit--recursiveThe --bos-limit option passes a given limit string to a BOS session. The --recursive
+option specifies a slot or other higher-level component in the limit string
sat bootprepThe --delete-ims-jobs option was added to sat bootprep run. It deletes IMS
+jobs after sat bootprep is run. Jobs are no longer deleted by default.
sat statussat status now includes information about nodes’ CFS configuration statuses, such
+as desired configuration, configuration status, and error count.
The output of sat status now splits different component types into different report tables.
The following options were added to sat status.
--hsm-fields, --sls-fields, --cfs-fields--bos-templateThe --hsm-fields, --sls-fields, --cfs-fields options limit the output columns
+according to specified CSM services.
The --bos-template option filters the status report according to the specified
+session template’s boot sets.
The following components were modified to be compatible with CSM 1.2.
+sat-cfs-install container image and Helm chartsat-install-utility container imageThe sat-ncn ansible role provided by sat-cfs-install was modified to enable
+GPG checks on packages while leaving GPG checks disabled on repository metadata.
Updated urllib3 dependency to version 1.26.5 to mitigate CVE-2021-33503 and refreshed +Python dependency versions.
+Minor bug fixes were made in each of the repositories. For full change lists, see each +repository’s CHANGELOG.md file.
+The known issues listed under the SAT 2.2 release were fixed.
+SAT 2.2.16 was released on February 25th, 2022.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container image and Helm chartIt also added the following new components:
+sat-install-utility container imagecfs-config-util container imageThe following sections detail the changes in this release.
+sat command unavailable in sat bash shellAfter launching a shell within the SAT container with sat bash, the sat command will not
+be found. For example:
(CONTAINER-ID) sat-container:~ # sat status
+bash: sat: command not found
+This can be resolved temporarily in one of two ways. /sat/venv/bin/ may be prepended to the
+$PATH environment variable:
(CONTAINER-ID) sat-container:~ # export PATH=/sat/venv/bin:$PATH
+(CONTAINER-ID) sat-container:~ # sat status
+Or, the file /sat/venv/bin/activate may be sourced:
(CONTAINER-ID) sat-container:~ # source /sat/venv/bin/activate
+(CONTAINER-ID) sat-container:~ # sat status
+sat bash shellAfter launching a shell within the SAT container with sat bash, tab completion for sat
+commands does not work.
This can be resolved temporarily by sourcing the file /etc/bash_completion.d/sat-completion.bash:
source /etc/bash_completion.d/sat-completion.bash
+sat in root directorysat commands will not work if the current directory is /. For example:
ncn-m001:/ # sat --help
+Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: open /dev/console: operation not permitted: OCI runtime permission denied error
+To resolve, run sat in another directory.
sat in config directorysat commands will not work if the current directory is ~/.config/sat. For example:
ncn-m001:~/.config/sat # sat --help
+Error: /root/.config/sat: duplicate mount destination
+To resolve, run sat in another directory.
sat commandssat bootprep automates the creation of CFS configurations, the build and
+customization of IMS images, and the creation of BOS session templates. See
+SAT Bootprep for details.sat slscheck performs a check for consistency between the System Layout
+Service (SLS) and the Hardware State Manager (HSM).sat bmccreds provides a simple interface for interacting with the System
+Configuration Service (SCSD) to set BMC Redfish credentials.sat hwhist displays hardware component history by xname (location) or by
+its Field-Replaceable Unit ID (FRUID). This command queries the Hardware
+State Manager (HSM) API to obtain this information. Since the sat hwhist
+command supports querying for the history of a component by its FRUID, the
+FRUID of components has been added to the output of sat hwinv.The following automation has been added to the install script, install.sh:
sat-config-import Kubernetes job, which is
+started when the sat-cfs-install Helm chart is deployed.The SAT product uploads additional information to the cray-product-catalog
+Kubernetes ConfigMap detailing the components it provides, including container
+(Docker) images, Helm charts, RPMs, and package repositories.
This information is used to support uninstall and activation of SAT product +versions moving forward.
+Beginning with the 2.2 release, SAT now provides partial support for the +uninstall and activation of the SAT product stream.
+See Uninstall: Removing a Version of SAT +and Activate: Switching Between Versions +for details.
+sat statusA Subrole column has been added to the output of sat status. This allows you
+to easily differentiate between master, worker, and storage nodes in the
+management role, for example.
Hostname information from SLS has been added to sat status output.
Support for JSON-formatted output has been added to commands which currently
+support the --format option, such as hwinv, status, and showrev.
Many usability improvements have been made to multiple sat commands,
+mostly related to filtering command output. The following are some highlights:
--fields option to display only specific fields for subcommands which
+display tabular reports.--filter queries
+so that the first match is used, similar to --sort-by.--filter, --fields, and --reverse for summaries
+displayed by sat hwinv.sat hwinv.The default log level for stderr has been changed from “WARNING” to “INFO”. For +details, see SAT Logging.
+With the command-line options --loglevel-stderr and --loglevel-file, the log level
+can now be configured separately for stderr and the log file.
The existing --loglevel option is now an alias for the --loglevel-stderr option.
The Podman wrapper script is the script installed at /usr/bin/sat on the
+master management NCNs by the cray-sat-podman RPM that runs the cray-sat
+container in podman. The following subsections detail improvements that were
+made to the wrapper script in this release.
The Podman wrapper script that launches the cray-sat container with podman
+has been modified to mount the user’s current directory and home directory into
+the cray-sat container to provide access to local files in the container.
The man page for the Podman wrapper script, which is accessed by typing man sat on a master management NCN, has been improved to document the following:
Fixed issues with redirecting stdout and stderr, and piping output to commands,
+such as awk, less, and more.
A new sat option has been added to configure the HTTP timeout length for
+requests to the API gateway. See sat-man sat for details.
sat bootsys ImprovementsMany improvements and fixes have been made to sat bootsys. The following are some
+highlights:
--excluded-ncns option, which can be used to omit NCNs
+from the platform-services and ncn-power stages in case they are
+inaccessible.sat bootsys shutdown now prompt the user to
+continue before proceeding. A new option, --disruptive, will bypass this.platform-services
+stage of sat bootsys boot.sat xname2nid Improvementssat xname2nid can now recursively expand slot, chassis, and cabinet xnames to
+a list of nids in those locations.
A new --format option has been added to sat xname2nid. It sets the output format to
+either “range” (the default) or “nid”. The “range” format displays nids in a
+compressed range format suitable for use with a workload manager like Slurm.
The commands which interact with HSM (e.g., sat status and sat hwinv) now
+use the v2 HSM API.
sat diag Limited to HSN Switchessat diag will now only operate against HSN switches by default. These are the
+only controllers that support running diagnostics with HMJTD.
sat showrev EnhancementsA column has been added to the output of sat showrev that indicates whether a
+product version is “active”. The definition of “active” varies across products,
+and not all products may set an “active” version.
For SAT, the active version is the one with its hosted-type package repository in
+Nexus set as the member of the group-type package repository in Nexus,
+meaning that it will be used when installing the cray-sat-podman RPM.
cray-sat Container Image Size ReductionThe size of the cray-sat container image has been approximately cut in half by
+leveraging multi-stage builds. This also improved the repeatability of the unit
+tests by running them in the container.
Minor bug fixes were made in cray-sat and in cray-sat-podman. For full change lists,
+see each repository’s CHANGELOG.md file.
We released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named “ncn-personalization”.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+executables on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and config file options +have been removed. For further instructions, see Remove Obsolete Configuration +File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes masters and workers.
sat firmware.cray-sat container image.sat firmware command.We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes management cluster (i.e., workers
+and masters). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Config File Location ChangeThe default location of the SAT config file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own config files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment variable.
Additionally, if a config file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies option.--list-node-accels option. The count of
+node accelerators is also included for each node.--list-node-accel-risers
+option. The count of node accelerator risers is also included for each node.--list-node-hsn-nics
+option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For detailed +instructions, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding config-file
+option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding config file options were
+deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchSee the System Admin Toolkit Command Overview +and the table of commands in the SAT Authentication section +of this document for more details on each of these commands.
+ + + + + +SAT provides an automated solution for creating CFS configurations, building +and configuring images in IMS, and creating BOS session templates based on a +given input file which defines how those configurations, images, and session +templates should be created.
+This automated process centers around the sat bootprep command. Man page
+documentation for sat bootprep can be viewed similarly to other SAT commands.
ncn-m001# sat-man sat-bootprep
+sat bootprep is used to create CFS configurations, build and
+rename IMS images, and create BOS session templates which tie the
+configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
+including (but not limited to) performing BOS operations (such as creating BOS
+sessions), powering on and off cabinets, and checking the state of the system
+prior to shutdown.
The input file provided to sat bootprep is a YAML-formatted file containing
+information which CFS, IMS, and BOS use to create configurations, images, and
+BOS session templates respectively. Writing and modifying these input files is
+the main task associated with using sat bootprep. An input file is composed of
+three main sections, one each for configurations, images, and session templates.
+These sections may be specified in any order, and any of the sections may be
+omitted if desired.
The configurations section begins with a configurations: key.
---
+configurations:
+Under this key, the user can list one or more configurations to create. For +each configuration, a name should be given, in addition to the list of layers +which comprise the configuration. Each layer can be defined by a product name +and optionally a version number, or commit hash or branch in the product’s +configuration repository. Alternatively, a layer can be defined by a Git +repository URL directly, along with an associated branch or commit hash.
+When a configuration layer is specified in terms of a product name, the layer +is created in CFS by looking up relevant configuration information (including +the configuration repository and commit information) from the +cray-product-catalog Kubernetes ConfigMap as necessary. A version may be +supplied, but if it is absent, the version is assumed to be the latest version +found in the cray-product-catalog.
+---
+configurations:
+- name: example-configuration
+ layers:
+ - name: example product
+ playbook: example.yml
+ product:
+ name: example
+ version: 1.2.3
+Alternatively, a configuration layer may be specified by explicitly referencing
+the desired configuration repository, along with the branch containing the
+intended version of the Ansible playbooks. A commit hash may be specified by replacing
+branch with commit.
...
+ - name: another example product
+ playbook: another-example.yml
+ git:
+ url: "https://vcs.local/vcs/another-example-config-management.git"
+ branch: main
+ ...
+When sat bootprep is run against an input file, a CFS configuration will be
+created corresponding to each configuration in the configurations section. For
+example, the configuration created from an input file with the layers listed
+above might look something like the following:
{
+ "lastUpdated": "2022-02-07T21:47:49Z",
+ "layers": [
+ {
+ "cloneUrl": "https://vcs.local/vcs/example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "example product",
+ "playbook": "example.yml"
+ },
+ {
+ "cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "another example product",
+ "playbook": "another-example.yml"
+ }
+ ],
+ "name": "example-configuration"
+}
+After specifying configurations, the user may add images to the input file
+which are to be built by IMS. To add an images section, the user should add
+an images key.
---
+configurations:
+ ... (omitted for brevity)
+images:
+Under the images key, the user may define one or more images to be created in
+a list. Each element of the list defines a separate IMS image to be built and/or
+configured. Images must contain a name, as well as an ims section containing a
+definition of the image to be built and/or configured. Images may be defined by
+an image recipe, or by a pre-built image. Recipes and pre-built images are
+referred to by their names or IDs in IMS. The ims section should also contain
+an is_recipe property, which indicates whether the name or ID refers to an
+image recipe or a pre-built image. Images may also optionally provide a text
+description of the image. This description is not stored or used by sat bootprep or any CSM services, but is useful for documenting images in the input
+file.
---
+configurations:
+ ... (omitted for brevity)
+images:
+- name: example-compute-image
+ description: >
+ An example compute node image for illustrative purposes.
+ ims:
+ name: example-compute-image-recipe
+ is_recipe: true
+- name: another-example-compute-image
+ description: >
+ Another example compute node image.
+ ims:
+ id: <IMS image UUID>
+ is_recipe: false
+Images may also contain a configuration property in their definition, which
+specifies a configuration with which to customize the built image prior to
+booting. If a configuration is specified, then configuration groups must also
+be specified using the configuration_group_names property.
---
+configurations:
+ ... (omitted for brevity)
+images:
+- name: example-compute-image
+ description: >
+ An example compute node image for illustrative purposes.
+ ims:
+ name: example-compute-image-recipe
+ is_recipe: true
+ configuration: example configuration
+ configuration_group_names:
+ - Compute
+BOS session templates are the final section of the input file, and are defined
+under the session_templates key.
---
+configurations:
+ ... (omitted for brevity)
+images:
+ ... (omitted for brevity)
+session_templates:
+Each session template is defined in terms of its name, an image, a
+configuration, and a set of parameters which can be used to configure the
+session. The name, image, and configuration are specified with their respective
+name, image, and configuration keys. bos_parameters may also be
+specified; currently, the only setting under bos_parameters that is supported
+is boot_sets, which can be used to define boot sets in the BOS session
+template. Each boot set is defined under its own property under boot_sets, and
+the value of each boot set can contain the following properties, all of
+which are optional:
kernel_parameters: the parameters passed to the kernel on the command linenetwork: the network over which the nodes will bootnode_list: nodes to add to the boot setnode_roles_groups: HSM roles to add to the boot setnode_groups: HSM groups to add to the boot setrootfs_provider: the root file system providerrootfs_provider_passthrough: parameters to add to the rootfs= kernel
+parameterThe properties listed previously are the same as the parameters that can be +specified directly through BOS boot sets. More information can be found in the +CSM documentation on session +templates. +Additional properties not listed are passed through to the BOS session template +as written.
+An example session template might look like the following:
+configurations:
+ ... (omitted for brevity)
+images:
+ ... (omitted for brevity)
+session_templates:
+- name: example-session-template
+ image: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_list: []
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+Putting together all of the previous input file sections, an example bootprep input +file might look something like the following.
+---
+configurations:
+- name: cos-config
+ layers:
+ - name: cos-integration-2.2.87
+ playbook: site.yml
+ product:
+ name: cos
+ version: 2.2.87
+ branch: integration
+ - name: cpe-integration-21.12.3
+ playbook: pe_deploy.yml
+ product:
+ name: cpe
+ version: 21.12.3
+ branch: integration
+ - name: slurm-master-1.1.1
+ playbook: site.yml
+ product:
+ name: slurm
+ version: 1.1.1
+ branch: master
+images:
+- name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ ims:
+ is_recipe: true
+ name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ configuration: cos-config
+ configuration_group_names:
+ - Compute
+session_templates:
+- name: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ image: cray-shasta-compute-sles15sp3.x86_64-2.2.35
+ configuration: cos-config
+ bos_parameters:
+ boot_sets:
+ compute:
+ kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
+ node_roles_groups:
+ - Compute
+It is possible to create an example bootprep input file using values from the
+system’s product catalog using the sat bootprep generate-example command.
ncn-m001# sat bootprep generate-example
+INFO: Using latest version (2.3.24-20220113160653) of product cos
+INFO: Using latest version (21.11.4) of product cpe
+INFO: Using latest version (1.0.7) of product slurm
+INFO: Using latest version (1.1.24) of product analytics
+INFO: Using latest version (2.1.5) of product uan
+INFO: Using latest version (21.11.4) of product cpe
+INFO: Using latest version (1.0.7) of product slurm
+INFO: Using latest version (1.1.24) of product analytics
+INFO: Using latest version (2.3.24-20220113160653) of product cos
+INFO: Using latest version (2.1.5) of product uan
+INFO: Wrote example bootprep input file to ./example-bootprep-input.yaml.
+This file should be reviewed and edited to match the desired parameters of the +configurations, images, and session templates.
+The contents of the YAML input files described above must conform to a schema +which defines the structure of the data. The schema definition is written using +the JSON Schema format. (Although the format is named “JSON Schema”, the schema +itself is written in YAML as well.) More information, including introductory +materials and a formal specification of the JSON Schema metaschema, can be found +on the JSON Schema website.
+To view the exact schema specification, run sat bootprep view-schema.
ncn-m001# sat bootprep view-schema
+---
+$schema: "https://json-schema.org/draft-07/schema"
+title: Bootprep Input File
+description: >
+ A description of the set of CFS configurations to create, the set of IMS
+ images to create and optionally customize with the defined CFS configurations,
+ and the set of BOS session templates to create that reference the defined
+ images and configurations.
+type: object
+additionalProperties: false
+properties:
+ ...
+The raw schema definition can be difficult to understand without experience +working with JSON Schema specifications. For this reason, a feature was included +which can generate user-friendly HTML documentation for the input file schema +which can be browsed with the user’s preferred web browser.
+Create a documentation tarball using sat bootprep.
ncn-m001# sat bootprep generate-docs
+INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
+An alternate output directory can be specified with the --output-dir
+option. The generated tarball is always named bootprep-schema-docs.tar.gz.
ncn-m001# sat bootprep generate-docs --output-dir /tmp
+INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
+From another machine, copy the tarball to a local directory.
+another-machine$ scp root@ncn-m001:bootprep-schema-docs.tar.gz .
+Extract the contents of the tarball and open the contained index.html.
another-machine$ tar xzvf bootprep-schema-docs.tar.gz
+x bootprep-schema-docs/
+x bootprep-schema-docs/index.html
+x bootprep-schema-docs/schema_doc.css
+x bootprep-schema-docs/schema_doc.min.js
+another-machine$ open bootprep-schema-docs/index.html
+Describes how to upgrade the System Admin Toolkit (SAT) product
+stream by using the Compute Node Environment (CNE) installer (cne-install).
+The CNE installer can be used only for upgrades and not for fresh installations.
+For installation instructions, see Install the System Admin Toolkit Product
+Stream.
Upgrading SAT with cne-install is recommended because the process is both
+automated and logged to help you save time. The CNE installer can be used to
+upgrade SAT alone or with other supported products. For more information
+on cne-install and its options, refer to the HPE Cray EX System Software
+Getting Started Guide (S-8000).
...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being upgraded.Start a typescript and set the shell prompt.
+The typescript will record the commands and the output from this upgrade. +The prompt is set to include the date and time.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Copy the release distribution gzipped tar file to ncn-m001.
The cne-install command installs all files in the media directory
+by default. If you are upgrading SAT alone, ensure only the SAT tarball is in
+the media directory.
Run the CNE installer.
+If you are upgrading SAT along with other supported products, run the +following command.
+ncn-m001# cne-install -m MEDIA_DIR install -B WORKING_BRANCH -bpc BOOTPREP_CONFIG_CN \
+ -bpn BOOTPREP_CONFIG_NCN
+The cne-install command will use the provided BOOTPREP_CONFIG_CN and
+BOOTPREP_CONFIG_NCN files for the run.
If you are upgrading SAT alone, run the following commands.
+ncn-m001# cne-install -m MEDIA_DIR install -B '{{product_type}}-{{version_x_y_z}}' \
+ -bpn BOOTPREP_CONFIG_NCN -e update_working_branches
+ncn-m001# cne-install -m MEDIA_DIR install -B '{{product_type}}-{{version_x_y_z}}' \
+ -bpn BOOTPREP_CONFIG_NCN -b sat_bootprep_ncn -e ncn_personalization
+Optional: Stop the typescript.
+NOTE: This step can be skipped if you wish to use the same typescript +for the remainder of the SAT upgrade (see Next Steps).
+ncn-m001# exit
+SAT version x.y.z is now upgraded, meaning the SAT x.y.z release
+has been loaded into the system software repository.
sat command is available.At this point, the release distribution files can be removed from the system as +described in Post-Upgrade Cleanup Procedure.
+If other HPE Cray EX software products are being upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +(S-8000) to determine which step +to execute next.
+If no other HPE Cray EX software products are being upgraded at this time, +execute the SAT Post-Upgrade procedures:
+ +Optional: Remove the SAT release distribution tar file and extracted directory.
+ncn-m001# rm sat-x.y.z.tar.gz
+ncn-m001# rm -rf sat-x.y.z/
+After upgrading SAT, if using the configuration file from a previous version,
+there may be configuration file sections no longer used in the new version.
+For example, when upgrading from Shasta 1.4 to Shasta 1.5, the [redfish]
+configuration file section is no longer used. In that case, the following
+warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve
+the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are +“unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accommodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
ncn-m001:~ # grep -A 3 logging ~/.config/sat/sat.toml
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checkssat sensorsThe following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xnamesat xname2nidsat swapThe SAT Grafana Dashboards display messages that are generated by the HSN (High Speed Network) and reported through +Redfish. The messages are displayed based on severity.
+Grafana can be accessed via web browser at the following URL:
+https://sma-grafana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Grafana being https://sma-grafana.cmn.EXAMPLE_DOMAIN.com
For more information on accessing the Grafana Dashboards, refer to Access the Grafana Monitoring UI in the +SMA product documentation.
+For more information on the interpretation of metrics for the SAT Grafana Dashboards, refer to “Fabric Telemetry +Kafka Topics” in the SMA product documentation.
+There are four Fabric Telemetry dashboards used in SAT that report on the HSN. Two contain chart panels and two display +telemetry in a tabular format.
+| Dashboard Name | +Display Type | +
|---|---|
| Fabric Congestion | +Chart Panels | +
| Fabric RFC3635 | +Chart Panels | +
| Fabric Errors | +Tabular Format | +
| Fabric Port State | +Tabular Format | +
The tabular format presents a single point of telemetry for a given location and metric, either because the telemetry +is not numerical or that it changes infrequently. The value shown is the most recently reported value for that location +during the time range selected, if any. The interval setting is not used for tabular dashboards.
+Shows the Interval and Locations Options for the available telemetry.
+
The value of the Interval option sets the time resolution of the received telemetry. This works a bit like a +histogram, with the available telemetry in an interval of time going into a “bucket” and averaging out to a single +point on the chart or table. The special value auto will choose an interval based on the time range selected.
+For more information, refer to Grafana Templates and Variables.
+The Locations option allows restriction of the telemetry shown by locations, either individual links or all links +in a switch. The selection presented updates dynamically according to time range, except for the errors dashboard, +which always has entries for all links and switches, although the errors shown are restricted to the selected time +range.
+The chart panels for the RFC3635 and Congestion dashboards allow selection of a single location from the chart’s legend +or the trace on the chart.
+
SAT Grafana Dashboards provide system administrators a way to view fabric telemetry data across all Rosetta switches in +the system and assess the past and present health of the high-speed network. It also allows the ability to drill down +to view data for specific ports on specific switches.
+This dashboard contains the variable, Port Type not found in the other dashboards. The possible values are edge, +local, and global and correspond to the link’s relationship to the network topology. The locations presented in the +panels are restricted to the values (any combination, defaults to “all”) selected.
+The metric values for links of a given port type are similar in value to each other but very distinct from the values of +other types. If the values for different port types are all plotted together, the values for links with lower values are +indistinguishable from zero when plotted.
+The port type of a link is reported as a port state “subtype” event when defined at port initialization.
+
This dashboard reports error counters in a tabular format in three panels.
+There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+Unlike other dashboards, the locations presented are all locations in the system rather than having telemetry within +the time range selected. However, the values are taken from telemetry within the time range.
+
There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+The Fabric Port State telemetry is distinct because it typically is not numeric. It also updates infrequently, so a +long time range may be necessary to obtain any values. Port State is refreshed daily, so a time range of 24 hours +results in all states for all links in the system being shown.
+The three columns named, group, switch, and port are not port state events, but extra information included with +all port state events.
+
For more information on performance counters, refer to +Definitions of Managed Objects for the Ethernet-like Interface Types, +an Internet standards document.
+Because these metrics are counters that only increase over time, the values plotted are the change in the counter’s +value over the interval setting.
+ + + + + +Kibana is an open source analytics and visualization platform designed to search, view, and interact with data stored +in Elasticsearch indices. Kibana runs as a web service and has a browser-based interface. It offers visual output of +node data in the forms of charts, tables and maps that display real-time Elasticsearch queries. Viewing system data in +this way breaks down the complexity of large data volumes into easily understood information.
+Kibana can be accessed via web browser at the following URL:
+https://sma-kibana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Kibana being https://sma-kibana.cmn.EXAMPLE_DOMAIN.com
For more information on accessing the Kibana Dashboards, refer to View Logs Via Kibana in the SMA product +documentation.
+Additional details about the AER, ATOM, Heartbeat, Kernel, MCE, and RAS Daemon Kibana Dashboards are included in this +table.
+| Dashboard | +Short Description | +Long Description | +Kibana Visualization and Search Name | +
|---|---|---|---|
sat-aer |
+AER corrected | +Corrected Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-corrected Search: sat-aer-corrected |
+
sat-aer |
+AER fatal | +Fatal Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-fatal Search: sat-aer-fatal |
+
sat-atom |
+ATOM failures | +Application Task Orchestration and Management tests are run on a node when a job finishes. Test failures are logged. | +sat-atom-failed |
+
sat-atom |
+ATOM admindown |
+Application Task Orchestration and Management test failures can result in nodes being marked admindown. An admindown node is not available for job launch. |
+sat-atom-admindown |
+
sat-heartbeat |
+Heartbeat loss events | +Heartbeat loss event messages reported by the hbtd pods that monitor for heartbeats across nodes in the system. |
+sat-heartbeat |
+
sat-kernel |
+Kernel assertions | +The kernel software performs a failed assertion when some condition represents a serious fault. The node goes down. | +sat-kassertions |
+
sat-kernel |
+Kernel panics | +The kernel panics when something is seriously wrong. The node goes down. | +sat-kernel-panic |
+
sat-kernel |
+Lustre bugs (LBUGs) | +The Lustre software in the kernel stack performs a failed assertion when some condition related to file system logic represents a serious fault. The node goes down. | +sat-lbug |
+
sat-kernel |
+CPU stalls | +CPU stalls are serous conditions that can reduce node performance, and sometimes cause a node to go down. Technically these are Read-Copy-Update stalls where software in the kernel stack holds onto memory for too long. Read-Copy-Update is a vital aspect of kernel performance and rather esoteric. | +sat-cpu-stall |
+
sat-kernel |
+Out of memory | +An Out Of Memory (OOM) condition has occurred. The kernel must kill a process to continue. The kernel will select an expendable process when possible. If there is no expendable process the node usually goes down in some manner. Even if there are expendable processes the job is likely to be impacted. OOM conditions are best avoided. | +sat-oom |
+
sat-mce |
+MCE | +Machine Check Exceptions (MCE) are errors detected at the processor level. | +sat-mce |
+
sat-rasdaemon |
+rasdaemon errors |
+Errors from the rasdaemon service on nodes. The rasdaemon service is the Reliability, Availability, and Serviceability Daemon, and it is intended to collect all hardware error events reported by the Linux kernel, including PCI and MCE errors. This may include certain HSN errors in the future. |
+sat-rasdaemon-error |
+
sat-rasdaemon |
+rasdaemon messages |
+All messages from the rasdaemon service on nodes. |
+sat-rasdaemon |
+
By default, search highlighting is enabled. This procedure instructs how to disable search highlighting.
+The Kibana Dashboard should be open on your system.
+Navigate to Management
+Navigate to Advanced Settings in the Kibana section, below the Elastic search section
+Scroll down to the Discover section
+Change Highlight results from on to off
+Click Save to save changes
+The AER Dashboard displays errors that come from the PCI Express Advanced Error Reporting (AER) driver. These errors +are split up into separate visualizations depending on whether they are fatal or corrected errors.
+Go to the dashboard section.
+Select sat-aer dashboard.
Choose the time range of interest.
+View the Corrected and Fatal Advanced Error Reporting messages from PCI Express devices on each node. View the +matching log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on +the left. If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass +next to each NID.
+The ATOM (Application Task Orchestration and Management) Dashboard displays node failures that occur during health
+checks and application test failures. Some test failures are of possible interest even though a node is not marked
+admindown or otherwise fails. They are of clear interest if a node is marked admindown, and might provide
+clues if a node otherwise fails. They might also show application problems.
HPE Cray EX is installed on the system along with the System Admin Toolkit, which contains the ATOM Kibana Dashboard.
+Go to the dashboard section.
+Select sat-atom dashboard.
Choose the time range of interest.
+View any nodes marked admindown and any ATOM test failures. These failures occur during health checks and
+application test failures. Test failures marked admindown are important to note. View the matching log messages
+in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. If desired,
+results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to each NID.
The Heartbeat Dashboard displays heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd
+pods are responsible for monitoring nodes in the system for heartbeat loss.
Go to the dashboard section.
+Select sat-heartbeat dashboard.
Choose the time range of interest.
+View the heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods are responsible
+for monitoring nodes in the system for heartbeat loss. View the matching log messages in the panel.
The Kernel Dashboard displays compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. +The messages reveal if Lustre has experienced a fatal error on any compute nodes in the system. A CPU stall is a serious +problem that might result in a node failure. Out-of-memory conditions can be due to applications or system problems and +may require expert analysis. They provide useful clues for some node failures and may reveal if an application is using +too much memory.
+Go to the dashboard section.
+Select sat-kernel dashboard.
Choose the time range of interest.
+View the compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. View the matching +log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. +If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to +each NID.
+The MCE Dashboard displays CPU detected processor-level hardware errors.
+Go to the dashboard section.
+Select sat-mce dashboard.
Choose the time range of interest.
+View the Machine Check Exceptions (MCEs) listed including the counts per NID (node). For an MCE, the CPU number and +DIMM number can be found in the message, if applicable. View the matching log messages in the panel(s) on the right, +and view the counts of each message per NID in the panel(s) on the left. If desired, results can be filtered by NID +by clicking the icon showing a + inside a magnifying glass next to each NID.
+The RAS Daemon Dashboard displays errors that come from the Reliability, Availability, and Serviceability (RAS) daemon
+service on nodes in the system. This service collects all hardware error events reported by the Linux kernel, including
+PCI and MCE errors. As a result there may be some duplication between the messages presented here and the messages
+presented in the MCE and AER dashboards. This dashboard splits up the messages into two separate visualizations, one
+for only messages of severity emerg or err and another for all messages from rasdaemon.
Go to the dashboard section.
+Select sat-rasdaemon dashboard.
Choose the time range of interest.
+View the errors that come from the Reliability, Availability, and Serviceability (RAS) daemon service on nodes in +the system. View the matching log messages in the panel(s) on the right, and view the counts of each message per NID +in the panel(s) on the left. If desired, results can be filtered by NID by clicking the icon showing a + inside +a magnifying glass next to each NID.
+Describes how to install or upgrade the System Admin Toolkit (SAT) product +stream.
+Ellipses (...) in shell output indicate omitted lines.
In the examples below, replace x.y.z with the version of the SAT product stream
+being installed.
‘manager’ and ‘master’ are used interchangeably in the steps below.
+To upgrade SAT, execute the pre-installation, installation, and post-installation +procedures for a newer distribution. The newly installed version will become +the default.
+In SAT 2.4, you can instead upgrade the product stream by using the +Compute Node Environment (CNE) installer. It is recommended that you upgrade +SAT with the CNE installer because the process is both automated and logged +to help you save time. For more information, see +SAT Upgrade with CNE Installer.
+Start a typescript and set the shell prompt.
+The typescript will record the commands and the output from this installation. +The prompt is set to include the date and time.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+Copy the release distribution gzipped tar file to ncn-m001.
Unzip and extract the release distribution.
+ncn-m001# tar -xvzf sat-x.y.z.tar.gz
+Change directory to the extracted release distribution directory.
+ncn-m001# cd sat-x.y.z
+Run the installer: install.sh.
The script produces a lot of output. A successful install ends with “SAT
+version x.y.z has been installed”, where x.y.z is the SAT product version.
ncn-m001# ./install.sh
+====> Installing System Admin Toolkit version x.y.z
+...
+====> Waiting 300 seconds for sat-config-import-x.y.z to complete
+...
+====> SAT version x.y.z has been installed.
+Optional: Stop the typescript.
+NOTE: This step can be skipped if you wish to use the same typescript +for the remainder of the SAT install (see Next Steps).
+ncn-m001# exit
+SAT version x.y.z is now installed/upgraded, meaning the SAT x.y.z release
+has been loaded into the system software repository.
sat command won’t be available until the NCN Personalization
+procedure has been executed.If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +(S-8000) to determine which step to +execute next.
+If no other HPE Cray EX software products are being installed or upgraded at this time, +proceed to the sections listed below.
+NOTE: The procedures in Configure SAT are only required during the +first installation of SAT. However, the NCN Personalization procedure +is required both when installing and upgrading SAT.
+If performing a fresh install, execute the Configure SAT procedures:
+ +Execute the NCN Personalization procedure:
+ +If performing an upgrade, execute the SAT Post-Upgrade procedures:
+ +NOTE: The Set System Revision Information procedure is not required +after upgrading from SAT 2.1 or later.
+Initially, as part of the installation and configuration, SAT authentication is set up so SAT commands can be used in
+later steps of the install process. The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For instructions on editing Role Mappings see
+Create Internal User Accounts in the Keycloak Shasta Realm in the CSM product documentation.
+For additional information on SAT authentication, see System Security and Authentication in the CSM
+documentation.
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Some SAT subcommands make requests to the Shasta services through the API gateway and thus require authentication to
+the API gateway in order to function. Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured (see: Generate SAT S3 Credentials). In order to use the SAT S3 bucket,
+the System Administrator must generate the S3 access key and secret keys and write them to a local file. This must be
+done on every Kubernetes manager node where SAT commands are run.
Below is a table describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bmccreds |
+Requires authentication to the API gateway. | +sat-bmccreds |
+Set BMC passwords. | +
sat bootprep |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is done on ncn-m001 during the install. |
+sat-bootprep |
+Prepare to boot nodes with images and configurations. | +
sat bootsys |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwhist |
+Requires authentication to the API gateway. | +sat-hwhist |
+Report hardware component history. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat k8s |
+Requires Kubernetes configuration and authentication, which is automatically configured on ncn-m001 during the install. |
+sat-k8s |
+Report on Kubernetes replica sets that have co-located (on the same node) replicas. | +
sat linkhealth |
++ | + | This command has been deprecated. | +
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node XNames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat slscheck |
+Requires authentication to the API gateway. | +sat-slscheck |
+Perform a cross-check between SLS and HSM. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC XNames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, you must run the sat auth command. This command will prompt for a password
+on the command line. The username value is obtained from the following locations, in order of higher precedence to lower
+precedence:
--username global command-line option.username option in the api_gateway section of the config file at ~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file will be obtained and saved to
+~/.config/sat/tokens. Subsequent sat commands will determine the username the same way as sat auth described above,
+and will use the token for that username if it has been obtained and saved by sat auth.
sat CLI has been installed following Install The System Admin Toolkit Product Stream.The following is the procedure to globally configure the username used by SAT and authenticate to the API gateway:
+Generate a default SAT configuration file, if one does not exist.
+ncn-m001# sat init
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the config file already exists, it will print out an error:
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway section of the config file. For
+example:
username = "crayadmin"
+Run sat auth. Enter your password when prompted. For example:
ncn-m001# sat auth
+Password for crayadmin:
+Succeeded!
+Other sat commands are now authenticated to make requests to the API gateway. For example:
ncn-m001# sat status
+Generate S3 credentials and write them to a local file so the SAT user can access S3 storage. In order to use the SAT +S3 bucket, the System Administrator must generate the S3 access key and secret keys and write them to a local file. +This must be done on every Kubernetes master node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the site-specific information set with sat setrev
+(see: Set System Revision Information).
NOTE: This procedure is only required after initially installing SAT. It is not +required after upgrading SAT.
+Ensure the files are readable only by root.
ncn-m001# touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+ncn-m001# chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+Get the SAT configuration file’s endpoint value.
+NOTE: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
ncn-m001# grep endpoint ~/.config/sat/sat.toml
+# endpoint = "https://rgw-vip.nmn"
+Get the sat-s3-credentials secret’s endpoint value.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, change the SAT configuration file’s endpoint value to match the secret’s.
+Copy SAT configurations to each manager node on the system.
+ncn-m001# for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+NOTE: Depending on how many manager nodes are on the system, the list of manager nodes may
+be different. This example assumes three manager nodes, where the configuration files must be
+copied from ncn-m001 to ncn-m002 and ncn-m003. Therefore, the list of hosts above is
+ncn-m002 and ncn-m003.
HPE service representatives use system revision information data to identify +systems in support cases.
+Set System Revision Information.
+Run sat setrev and follow the prompts to set the following site-specific values:
TIP: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. In other words, “System type” is EX-1C.
+ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Verify System Revision Information.
+Run sat showrev and verify the output shown in the “System Revision Information table.”
The following example shows sample table output.
+ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+A new CFS configuration layer must be added to the CFS configuration used on +management NCNs. It is required following SAT installation and configuration. +This procedure describes how to add that layer.
+...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being installed.Start a typescript if not already using one, and set the shell prompt.
+The typescript will record the commands and the output from this installation. +The prompt is set to include the date and time.
+ncn-m001# script -af product-sat.$(date +%Y-%m-%d).txt
+ncn-m001# export PS1='\u@\H \D{%Y-%m-%d} \t \w # '
+The SAT release distribution includes a script, update-mgmt-ncn-cfs-config.sh,
+that updates a CFS configuration to include the SAT layer required to
+install and configure SAT on the management NCNs.
The script supports modifying a named CFS configuration in CFS, a CFS +configuration defined in a JSON file, or the CFS configuration +currently applied to particular components in CFS.
+The script also includes options for specifying:
+This procedure is split into three alternatives, which cover common use cases:
+If none of these alternatives fit your use case, see Advanced Options for +Updating CFS Configurations.
+Use this alternative if there is already a CFS configuration assigned to the +management NCNs and you would like to update it in place for the new version of +SAT.
+Run the script with the following options:
+ncn-m001# ./update-mgmt-ncn-cfs-config.sh --base-query role=Management,type=Node --save
+Examine the output to ensure the CFS configuration was updated.
+For example, if there is a single CFS configuration that applies to NCNs, and if +that configuration does not have a layer yet for any version of SAT, the +output will look like this:
+====> Updating CFS configuration(s)
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, ..., x3000c0s9b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s1b0n0
+...
+INFO: Found configuration "ncn-personalization" for component x3000c0s9b0n0
+...
+INFO: No layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml found.
+INFO: Adding a layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml to the end.
+INFO: Successfully saved CFS configuration "ncn-personalization"
+INFO: Successfully saved 1 changed CFS configurations.
+====> Completed CFS configuration(s)
+====> Cleaning up install dependencies
+Alternatively, if the CFS configuration already contains a layer for +SAT that just needs to be updated, the output will look like this:
+====> Updating CFS configuration(s)
+INFO: Querying CFS configurations for the following NCNs: x3000c0s1b0n0, ..., x3000c0s9b0n0
+INFO: Found configuration "ncn-personalization" for component x3000c0s1b0n0
+...
+INFO: Found configuration "ncn-personalization" for component x3000c0s9b0n0
+...
+INFO: Updating existing layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml
+INFO: Property "commit" of layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml updated from 01ae28c92b9b4740e9e0e01ae01216c6c2d89a65 to bcbd6db0803cc4137c7558df9546b0faab303cbd
+INFO: Property "name" of layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml updated from sat-2.2.16 to sat-sat-ncn-bcbd6db-20220608T170152
+INFO: Successfully saved CFS configuration "ncn-personalization"
+INFO: Successfully saved 1 changed CFS configurations.
+====> Completed CFS configuration(s)
+====> Cleaning up install dependencies
+Use this alternative if you are constructing a new CFS configuration for +management NCNs in a JSON file.
+Run the script with the following options, where JSON_FILE is an
+environment variable set to the path of the JSON file to modify:
ncn-m001# ./update-mgmt-ncn-cfs-config.sh --base-file $JSON_FILE --save
+Examine the output to ensure the JSON file was updated.
+For example, if the configuration defined in the JSON file does not have a layer yet for any +version of SAT, the output will look like this:
+====> Updating CFS configuration(s)
+INFO: No layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml found.
+INFO: Adding a layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml to the end.
+INFO: Successfully saved 1 changed CFS configurations.
+====> Completed CFS configuration(s)
+====> Cleaning up install dependencies
+Use this alternative if you are updating a specific named CFS configuration. +This may be the case if you are constructing a new CFS configuration during an +install or upgrade of multiple products.
+Run the script with the following options, where CFS_CONFIG_NAME is an
+environment variable set to the name of the CFS configuration to update.
ncn-m001# ./update-mgmt-ncn-cfs-config.sh --base-config $CFS_CONFIG_NAME --save
+Examine the output to ensure the CFS configuration was updated.
+For example, if the CFS configuration does not have a layer yet for any version of SAT, +the output will look like this:
+====> Updating CFS configuration(s)
+INFO: No layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml found.
+INFO: Adding a layer with repo path /vcs/cray/sat-config-management.git and playbook sat-ncn.yml to the end.
+INFO: Successfully saved CFS configuration "CFS_CONFIG_NAME"
+INFO: Successfully saved 1 changed CFS configurations.
+====> Completed CFS configuration(s)
+====> Cleaning up install dependencies
+If none of the alternatives described in the previous sections apply, view the
+full description of the options accepted by the update-mgmt-ncn-cfs-config.sh
+script by invoking it with the --help option.
ncn-m001# ./update-mgmt-ncn-cfs-config.sh --help
+After the CFS configuration that applies to management NCNs has been updated as +described in the Procedure to Update CFS Configuration, +execute the following steps to ensure the modified CFS configuration is re-applied to the management NCNs.
+Set an environment variable that refers to the name of the CFS configuration +to be applied to the management NCNs.
+ncn-m001# export CFS_CONFIG_NAME="ncn-personalization"
+Note: If the Update Active CFS Configuration +section was followed above, the name of the updated CFS configuration will +have been logged in the following format. If multiple CFS configurations +were modified, any one of them can be used in this procedure.
+INFO: Successfully saved CFS configuration "ncn-personalization"
+Obtain the name of the CFS configuration layer for SAT and save it in an +environment variable:
+ncn-m001# export SAT_LAYER_NAME=$(cray cfs configurations describe $CFS_CONFIG_NAME --format json \
+ | jq -r '.layers | map(select(.cloneUrl | contains("sat-config-management.git")))[0].name')
+Create a CFS session that executes only the SAT layer of the given CFS +configuration.
+The --configuration-limit option limits the configuration session to run
+only the SAT layer of the configuration.
ncn-m001# cray cfs sessions create --name "sat-session-${CFS_CONFIG_NAME}" --configuration-name \
+ "${CFS_CONFIG_NAME}" --configuration-limit "${SAT_LAYER_NAME}"
+Monitor the progress of the CFS session.
+Set an environment variable to name of the Ansible container within the pod +for the CFS session:
+ncn-m001# export ANSIBLE_CONTAINER=$(kubectl get pod -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME} -o json \
+ -o json | jq -r '.items[0].spec.containers | map(select(.name | contains("ansible"))) | .[0].name')
+Next, get the logs for the Ansible container.
+ncn-m001# kubectl logs -c $ANSIBLE_CONTAINER --tail 100 -f -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME}
+Ansible plays, which are run by the CFS session, will install SAT on all the +master management NCNs on the system. A summary of results can be found at +the end of the log output. The following example shows a successful session.
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+NOTE: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify that SAT was successfully configured.
+If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
NOTE: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
ncn-m001# sat --version
+sat 3.7.0
+NOTE: Upon first running sat, you may see additional output while the sat
+container image is downloaded. This will occur the first time sat is run on
+each manager NCN. For example, if you run sat for the first time on ncn-m001
+and then for the first time on ncn-m002, you will see this additional output
+both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+Stop the typescript.
+ncn-m001# exit
+SAT version x.y.z is now installed and configured:
The previous procedure is not always necessary because the CFS Batcher service +automatically detects configuration changes and will automatically create new +sessions to apply configuration changes according to certain rules. For more +information on these rules, refer to Configuration Management with +the CFS Batcher in the Cray System Management Documentation.
+The main scenario in which the CFS batcher will not automatically re-apply the +SAT layer is when the commit hash of the sat-config-management git repository +has not changed between SAT versions. The previous procedure ensures the +configuration is re-applied in all cases, and it is harmless if the batcher has +already applied an updated configuration.
+At this point, the release distribution files can be removed from the system as +described in Post-Installation Cleanup Procedure.
+If other HPE Cray EX software products are being installed or upgraded in conjunction +with SAT, refer to the HPE Cray EX System Software Getting Started Guide +(S-8000) to determine which step +to execute next.
+If no other HPE Cray EX software products are being installed at this time, +the installation process is complete. If no other HPE Cray EX software products +are being upgraded at this time, proceed to the remaining SAT Post-Upgrade +procedures:
+ +NOTE: The Set System Revision Information procedure is not required after upgrading from SAT 2.1 or later.
+Optional: Remove the SAT release distribution tar file and extracted directory.
+ncn-m001# rm sat-x.y.z.tar.gz
+ncn-m001# rm -rf sat-x.y.z/
+After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
+In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accommodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
ncn-m001:~ # grep -A 3 logging ~/.config/sat/sat.toml
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checkssat sensorsThe following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xnamesat xname2nidsat swapThis procedure can be used to uninstall a version of SAT.
+prodmgr.prodmgr command is available.Use sat showrev to list versions of SAT.
NOTE: It is not recommended to uninstall a version designated as “active”. +If the active version is uninstalled, then the activate procedure must be executed +on a remaining version.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+-------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+-------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+-------------------+-----------------------+
+Use prodmgr to uninstall a version of SAT.
This command will do three things:
+cray-product-catalog Kubernetes ConfigMap, so that it will no longer show up
+in the output of sat showrev.ncn-m001# prodmgr uninstall sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 has been removed.
+Removed Docker image cray/cray-sat:3.9.0
+Removed Docker image cray/sat-cfs-install:1.0.2
+Removed Docker image cray/sat-install-utility:1.4.0
+Deleted sat-2.2.10 from product catalog.
+This procedure can be used to downgrade the active version of SAT.
+prodmgr command is available.Use sat showrev to list versions of SAT.
ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | True | - | - |
+| sat | 2.2.10 | False | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Use prodmgr to activate a different version of SAT.
This command will do three things:
+2.2.10
+sets the repository sat-2.2.10-sle-15sp2 as the only member of the sat-sle-15sp2 group.2.2.10 as active within the product catalog, so that it appears active in the output of
+sat showrev.ncn-personalization). Specifically, it will ensure that the layer refers to the version of SAT CFS
+configuration content associated with the version of SAT being activated.ncn-m001# prodmgr activate sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 is now the default in sat-sle-15sp2.
+Set sat-2.2.10 as active in product catalog.
+Updated CFS configurations: [ncn-personalization]
+Verify that the chosen version is marked as active.
+ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------+--------------------+-----------------------+
+| product_name | product_version | active | images | image_recipes |
++--------------+-----------------+--------+--------------------+-----------------------+
+| sat | 2.3.3 | False | - | - |
+| sat | 2.2.10 | True | - | - |
++--------------+-----------------+--------+--------------------+-----------------------+
+Apply the modified CFS configuration to the management NCNs.
+At this point, Nexus package repositories have been modified to set a +particular package repository as active, but the SAT package may not have +been updated on management NCNs.
+To ensure that management NCNs have been updated to use the active SAT
+version, follow the Procedure to Apply CFS Configuration.
+Refer to the output from the prodmgr activate command to find the name of
+the modified CFS configuration. If more than one CFS configuration was
+modified, use the first one.
SAT can optionally be installed and configured on an external system to interact with CSM over the CAN.
+Most SAT subcommands work by accessing APIs which are reachable via the CAN. However, certain SAT commands depend on +host-based functionality on the management NCNs and will not work from an external system. This includes the following:
+platform-services and ncn-power stages of sat bootsys--local option of sat showrevInstalling SAT on an external system is not an officially supported configuration. These instructions are provided +“as-is” with the hope that they can useful for users who desire additional flexibility.
+Certain additional steps may need to be taken to install and configure SAT depending on the configuration of the +external system in use. These additional steps may include provisioning virtual machines, installing packages, or +configuring TLS certificates, and these steps are outside the scope of this documentation. This section covers only the +steps needed to configure SAT to use externally-accessible API endpoints exposed by CSM.
+kubectl, openssh, git, and curl are installed on the external system.Create a Python virtual environment.
+$ SAT_VENV_PATH="$(pwd)/venv"
+$ python3 -m venv ${SAT_VENV_PATH}
+$ . ${SAT_VENV_PATH}/bin/activate
+Clone the SAT source code.
+Note: To use SAT version 3.19, this example clones the release/3.19 branch of
+Cray-HPE/sat. However, for better clarity, these instructions include steps that apply only to
+versions newer than 3.19. Specifically, the instructions include references to the
+csm-api-client package, which was not a dependency of SAT in version 3.19.
(venv) $ git clone --branch=release/3.19 https://github.com/Cray-HPE/sat.git
+Set up the SAT CSM Python dependencies to be installed from their source code.
+SAT CSM Python dependency packages are not currently distributed publicly as
+source packages or binary distributions. They must be installed from
+their source code hosted on GitHub. Also, to install the cray-product-catalog
+Python package, you must first clone it locally. Use the following steps to
+modify the SAT CSM Python dependencies so they can be installed from their source code.
Clone the source code for cray-product-catalog.
(venv) $ git clone --branch v1.6.0 https://github.com/Cray-HPE/cray-product-catalog
+In the cray-product-catalog directory, create a file named .version
+that contains the version of cray-product-catalog.
(venv) $ echo 1.6.0 > cray-product-catalog/.version
+Open the “locked” requirements file in a text editor.
+(venv) $ vim sat/requirements.lock.txt
+Update the line containing cray-product-catalog so that it reflects the local path
+to cray-product-catalog.
It should read as follows:
+./cray-product-catalog
+For versions of SAT newer than 3.19, change the line containing csm-api-client to
+read as follows:
csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+(Optional) Confirm that requirements.lock.txt is modified as expected.
Note: For versions newer than 3.19, you will see both cray-product-catalog and csm-api-client.
+For version 3.19 and older, you will only see cray-product-catalog.
(venv) $ grep -E 'cray-product-catalog|csm-api-client' sat/requirements.lock.txt
+./cray-product-catalog
+csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+Install the modified SAT dependencies.
+(venv) $ pip install -r sat/requirements.lock.txt
+...
+Install the SAT Python package.
+(venv) $ pip install ./sat
+...
+Optional: Add the sat virtual environment to the user’s PATH environment variable.
If a shell other than bash is in use, replace ~/.bash_profile with the appropriate profile path.
If the virtual environment is not added to the user’s PATH environment variable, then
+source ${SAT_VENV_PATH}/bin/activate will need to be run before running any SAT commands.
(venv) $ deactivate
+$ echo export PATH=\"${SAT_VENV_PATH}/bin:${PATH}\" >> ~/.bash_profile
+$ source ~/.bash_profile
+Copy the file /etc/kubernetes/admin.conf from ncn-m001 to ~/.kube/config on the external system.
Note that this file contains credentials to authenticate against the Kubernetes API as the administrative user, so +it should be treated as sensitive.
+$ mkdir -p ~/.kube
+$ scp ncn-m001:/etc/kubernetes/admin.conf ~/.kube/config
+admin.conf 100% 5566 3.0MB/s 00:00
+Add a new entry for the hostname kubernetes to the external system’s /etc/hosts file.
The kubernetes hostname should correspond to the CAN IP address on ncn-m001. On CSM 1.2, this can be determined
+by querying the IP address of the bond0.cmn0 interface.
$ ssh ncn-m001 ip addr show bond0.cmn0
+13: bond0.cmn0@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.11/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+$ IP_ADDRESS=10.102.1.11
+On CSM versions prior to 1.2, the CAN IP can be determined by querying the IP address of the vlan007 interface.
$ ssh ncn-m001 ip addr show vlan007
+13: vlan007@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.10/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+$ IP_ADDRESS=10.102.1.10
+Once the IP address is determined, add an entry to /etc/hosts mapping the IP address to the hostname kubernetes.
$ echo "${IP_ADDRESS} kubernetes" | sudo tee -a /etc/hosts
+10.102.1.11 kubernetes
+Modify ~/.kube/config to set the cluster server address.
The value of the server key for the kubernetes cluster under the clusters section should be set to
+https://kubernetes:6443.
---
+clusters:
+- cluster:
+ certificate-authority-data: REDACTED
+ server: https://kubernetes:6443
+ name: kubernetes
+...
+Confirm that kubectl can access the CSM Kubernetes cluster.
$ kubectl get nodes
+NAME STATUS ROLES AGE VERSION
+ncn-m001 Ready master 135d v1.19.9
+ncn-m002 Ready master 136d v1.19.9
+ncn-m003 Ready master 136d v1.19.9
+ncn-w001 Ready <none> 136d v1.19.9
+ncn-w002 Ready <none> 136d v1.19.9
+ncn-w003 Ready <none> 136d v1.19.9
+Use sat init to create a configuration file for SAT.
$ sat init
+INFO: Configuration file "/home/user/.config/sat/sat.toml" generated.
+Copy the platform CA certificates from the management NCN and configure the certificates for use with SAT.
+If a shell other than bash is in use, replace ~/.bash_profile with the appropriate profile path.
$ scp ncn-m001:/etc/pki/trust/anchors/platform-ca-certs.crt .
+$ echo export REQUESTS_CA_BUNDLE=\"$(realpath platform-ca-certs.crt)\" >> ~/.bash_profile
+$ source ~/.bash_profile
+Edit the SAT configuration file to set the API and S3 hostnames.
+Externally available API endpoints are given domain names in PowerDNS, so the endpoints in the configuration file
+should each be set to subdomain.system-name.site-domain, where system-name and site-domain are replaced with
+the values specified during csi config init, and subdomain is the DNS name for the externally available service.
+For more information, refer to Externally Exposed Services in the Cray System Management Documentation.
The API gateway has the subdomain api, and S3 has the subdomain s3. The S3 endpoint runs on port 8080. The
+following options should be set in the SAT configuration file:
[api_gateway]
+host = "api.system-name.site-domain"
+
+[s3]
+endpoint = "http://s3.system-name.site-domain:8080"
+Edit the SAT configuration file to specify the Keycloak user which will be accessing the REST API.
+[api_gateway]
+username = "user"
+Authenticate against the API gateway with sat auth.
For more information, see SAT Authentication.
+Generate S3 credentials.
+For more information, see Generate SAT S3 Credentials.
+The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands
+used on the Cray XC platform. For more information on SAT commands, see System Admin Toolkit Command Overview.
Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
+Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and +are reported through Redfish.
+In CSM 1.3 and newer, the sat command is automatically available on all the
+Kubernetes NCNs. For more information, see SAT in CSM. Older
+versions of CSM do not have the sat command automatically available, and SAT
+must be installed as a separate product.
Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the
+HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are
+similarities between SAT commands and xt commands used on the Cray XC platform.
The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using Podman, a daemonless container runtime. SAT runs on
+Kubernetes manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
+sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
+(CONTAINER-ID)sat-container# sat status
+ncn-m001# sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
+example.
ncn-m001# sat-man status
+A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
+ncn-m001# man sat-podman
+The host name in a command prompt indicates where the command must be run. The account that must run the command is +also indicated in the prompt.
+root or super-user account always has the # character at the end of the prompt and has the host name of the
+host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
+referred to as user.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run on one of the Kubernetes Manager servers. (Non-interactive) | +
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
Examples of the sat status command used by an administrator:
ncn-m001# sat status
+ncn-m001# sat bash
+(CONTAINER_ID) sat-container# sat status
+In CSM 1.3 and newer, the sat command is automatically available on all the Kubernetes NCNs, but it is still possible
+to install SAT as a separate product stream. Any version of SAT installed as a separate product stream overrides the
+sat command available in CSM. Installing the SAT product stream allows additional supporting components to be added:
An entry for SAT in the cray-product-catalog Kubernetes ConfigMap is only created by installing the SAT product
+stream. Otherwise, there will be no entry for this version of SAT in the output of sat showrev.
The sat-install-utility container image is only available with the full SAT product stream. This container image
+provides uninstall and activate functionality when used with the prodmgr command. (In SAT 2.3 and older, SAT was
+only available to install as a separate product stream. Because these versions were packaged with
+sat-install-utility, it is still possible to uninstall these versions of SAT.)
The docs-sat RPM package is only available with the full SAT product stream.
The sat-config-management git repository in Gitea (VCS) and thus the SAT layer of NCN CFS configuration is
+only available with the full SAT product stream.
If the SAT product stream is not installed, there will be no configuration content for SAT in VCS. Therefore, CFS
+configurations that apply to NCNs (for example, ncn-personalization) should not include a SAT layer.
The SAT configuration layer modifies the permissions of files left over from prior installations of SAT, so that the
+Keycloak username that authenticates to the API gateway cannot be read by users other than root. Specifically, it
+it does the following:
Modifies the sat.toml configuration file which contains the username so that it is only readable by root.
Modifies the /root/.config/sat/tokens directory so that the directory is only readable by root. This is needed
+because the names of the files within the tokens directory contain the username.
Regardless of the SAT configuration being applied, passwords and the contents of the tokens are never readable by other +users. These permission changes only apply to files created by previous installations of SAT. In the current version of +SAT all files and directories are created with the appropriate permissions.
+Most sat subcommands depend on services or components from other products in the
+HPE Cray EX (Shasta) software stack. The following list shows these dependencies
+for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
+sat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nidThe 2.4.13 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.cfs-config-util container image.Because of installation refactoring efforts, the following two components +are no longer delivered with SAT:
+sat-cfs-install container imagesat-cfs-install Helm chartA version of the cray-sat container image is now included in CSM. For more
+information, see SAT in CSM.
The SAT install.sh script no longer uses a sat-cfs-install Helm chart and
+container image to upload its Ansible content to the sat-config-management
+repository in VCS. Instead, it uses Podman to run the cf-gitea-import container
+directly. Some of the benefits of this change include the following:
cray-sat container image and cray-sat-podman packagecray-sat Container Image and cray-sat-podman PackageIn older SAT releases, the sat wrapper script that was provided by the
+cray-sat-podman package installed on Kubernetes master NCNs included a
+hard-coded version of the cray-sat container image. As a result, every new
+version of the cray-sat image required a corresponding new version of the
+cray-sat-podman package.
In this release, this tight coupling of the cray-sat-podman package and the
+cray-sat container image was removed. The sat wrapper script provided
+by the cray-sat-podman package now looks for the version of the cray-sat
+container image in the /opt/cray/etc/sat/version file. This file is populated
+with the correct version of the cray-sat container image by the SAT layer of
+the CFS configuration that is applied to management NCNs. If the version file
+does not exist, the wrapper script defaults to the version of the cray-sat
+container image delivered with the latest version of CSM installed on the system.
The steps for performing NCN personalization as part of the SAT installation
+were moved out of the install.sh script and into a new
+update-mgmt-ncn-cfs-config.sh script that is provided in the SAT release
+distribution. The new script provides additional flexibility in how it modifies
+the NCN personalization CFS configuration for SAT. It can modify an existing CFS
+configuration by name, a CFS configuration being built in a JSON file, or an
+existing CFS configuration that applies to certain components. For more information,
+see Perform NCN Personalization.
sat bootprep FeaturesThe following new features were added to the sat bootprep command:
Variable substitutions using Jinja2 templates in certain fields of the
+sat bootprep input file
For more information, see +HPC CSM Software Recipe Variable Substitutions +and Dynamic Variable Substitutions.
+Schema version validation in the sat bootprep input files
For more information, see Providing a Schema Version.
+Ability to look up images and recipes provided by products
+For more information, see Defining IMS Images.
+The schema of the sat bootprep input files was also changed to support these
+new features:
base key instead of under an ims key. The old ims
+key is deprecated.base.image_ref.
+You should no longer use the IMS name of the image on which it depends.image.ims.name, image.ims.id, or image.image_ref. Specifying a string
+value directly under the image key is deprecated.For more information on defining IMS images and BOS session templates in the
+sat bootprep input file, see Defining IMS Images
+and Defining BOS Session Templates.
sat swapThe sat swap command was updated to support swapping compute and UAN blades
+with sat swap blade. This functionality is described in the following processes
+of the Cray System Management Documentation:
v2A new v2 version of the Boot Orchestration Service (BOS) is available in CSM
+1.3.0. SAT has added support for BOS v2. This impacts the following commands
+that interact with BOS:
sat bootprepsat bootsyssat statusBy default, SAT uses BOS v1. However, you can choose the BOS version you want
+to use. For more information, see Change the BOS Version.
sat statusWhen using BOS v2, sat status outputs additional fields. These fields show
+the most recent BOS session, session template, booted image, and boot status for
+each node. An additional --bos-fields option was added to limit the output of
+sat status to these fields. The fields are not displayed when using BOS v1.
This is the first release of SAT built from open source code repositories. +As a result, build infrastructure was changed to use an external Jenkins instance, +and artifacts are now published to an external Artifactory instance. These +changes should not impact the functionality of the SAT product in any way.
+paramiko Python package version was updated from 2.9.2 to 2.10.1 to
+mitigate CVE-2022-24302.oauthlib Python package version was updated from 3.2.0 to 3.2.1 to
+mitigate CVE-2022-36087.SAT stores information used to authenticate to the API gateway with Keycloak.
+Token files are stored in the ~/.config/sat/tokens/ directory. Those files
+have always had permissions appropriately set to restrict them to be readable
+only by the user.
Keycloak usernames used to authenticate to the API gateway are stored in the
+SAT config file at /.config/sat/sat.toml. Keycloak usernames are also used in
+the file names of tokens stored in /.config/sat/tokens. As an additional
+security measure, SAT now restricts the permissions of the SAT config file
+to be readable and writable only by the user. It also restricts the tokens
+directory and the entire SAT config directory ~/.config/sat to be accessible
+only by the user. This prevents other users on the system from viewing
+Keycloak usernames used to authenticate to the API gateway.
sat init did not print a message confirming a new
+configuration file was created.sat showrev exited with a traceback if the file
+/opt/cray/etc/site_info.yaml existed but was empty. This could occur if the
+user exited sat setrev with Ctrl-C.sat bootsys man page, and added a
+description of the command stages.The 2.3.4 version of the SAT product includes:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container imagesat-cfs-install Helm chartsat-install-utility container imagecfs-config-util container imagesat CommandsNone.
+When running sat commands, the current working directory is now mounted in the
+container as /sat/share, and the current working directory within the container
+is also /sat/share.
Files in the current working directory must be specified using relative paths to
+that directory, because the current working directory is always mounted on /sat/share.
+Absolute paths should be avoided, and paths that are outside of $HOME or $PWD
+are never accessible to the container environment.
The home directory is still mounted on the same path inside the container as it +is on the host.
+sat bootsysThe following options were added to sat bootsys.
--bos-limit--recursiveThe --bos-limit option passes a given limit string to a BOS session. The --recursive
+option specifies a slot or other higher-level component in the limit string
sat bootprepThe --delete-ims-jobs option was added to sat bootprep run. It deletes IMS
+jobs after sat bootprep is run. Jobs are no longer deleted by default.
sat statussat status now includes information about nodes’ CFS configuration statuses, such
+as desired configuration, configuration status, and error count.
The output of sat status now splits different component types into different report tables.
The following options were added to sat status.
--hsm-fields, --sls-fields, --cfs-fields--bos-templateThe --hsm-fields, --sls-fields, --cfs-fields options limit the output columns
+according to specified CSM services.
The --bos-template option filters the status report according to the specified
+session template’s boot sets.
The following components were modified to be compatible with CSM 1.2.
+sat-cfs-install container image and Helm chartsat-install-utility container imageThe sat-ncn Ansible role provided by sat-cfs-install was modified to enable
+GPG checks on packages while leaving GPG checks disabled on repository metadata.
Updated urllib3 dependency to version 1.26.5 to mitigate CVE-2021-33503 and refreshed
+Python dependency versions.
Minor bug fixes were made in each of the repositories. For full change lists,
+refer to each repository’s CHANGELOG.md file.
The known issues listed under the SAT 2.2 release were fixed.
+SAT 2.2.16 was released on February 25th, 2022.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container image and Helm chartIt also added the following new components:
+sat-install-utility container imagecfs-config-util container imageThe following sections detail the changes in this release.
+sat Command Unavailable in sat bash ShellAfter launching a shell within the SAT container with sat bash, the sat command will not
+be found. For example:
(CONTAINER-ID) sat-container:~ # sat status
+bash: sat: command not found
+This can be resolved temporarily in one of two ways. /sat/venv/bin/ may be prepended to the
+$PATH environment variable:
(CONTAINER-ID) sat-container:~ # export PATH=/sat/venv/bin:$PATH
+(CONTAINER-ID) sat-container:~ # sat status
+Or, the file /sat/venv/bin/activate may be sourced:
(CONTAINER-ID) sat-container:~ # source /sat/venv/bin/activate
+(CONTAINER-ID) sat-container:~ # sat status
+sat bash ShellAfter launching a shell within the SAT container with sat bash, tab completion for sat
+commands does not work.
This can be resolved temporarily by sourcing the file /etc/bash_completion.d/sat-completion.bash:
source /etc/bash_completion.d/sat-completion.bash
+sat in Root Directorysat commands will not work if the current directory is /. For example:
ncn-m001:/ # sat --help
+Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: open /dev/console: operation not permitted: OCI runtime permission denied error
+To resolve, run sat in another directory.
sat in Config Directorysat commands will not work if the current directory is ~/.config/sat. For example:
ncn-m001:~/.config/sat # sat --help
+Error: /root/.config/sat: duplicate mount destination
+To resolve, run sat in another directory.
sat Commandssat bootprep automates the creation of CFS configurations, the build and
+customization of IMS images, and the creation of BOS session templates. For
+more information, see SAT Bootprep.sat slscheck performs a check for consistency between the System Layout
+Service (SLS) and the Hardware State Manager (HSM).sat bmccreds provides a simple interface for interacting with the System
+Configuration Service (SCSD) to set BMC Redfish credentials.sat hwhist displays hardware component history by XName (location) or by
+its Field-Replaceable Unit ID (FRUID). This command queries the Hardware
+State Manager (HSM) API to obtain this information. Since the sat hwhist
+command supports querying for the history of a component by its FRUID, the
+FRUID of components has been added to the output of sat hwinv.The following automation has been added to the install script, install.sh:
sat-config-import Kubernetes job, which is
+started when the sat-cfs-install Helm chart is deployed.ncn-personalization).The SAT product uploads additional information to the cray-product-catalog
+Kubernetes ConfigMap detailing the components it provides, including container
+(Docker) images, Helm charts, RPMs, and package repositories.
This information is used to support uninstall and activation of SAT product +versions moving forward.
+Beginning with the 2.2 release, SAT now provides partial support for the +uninstall and activation of the SAT product stream.
+For more information, see Uninstall: Removing a Version of +SAT and Activate: Switching +Between Versions.
+sat statusA Subrole column has been added to the output of sat status. This allows you
+to easily differentiate between master, worker, and storage nodes in the
+management role, for example.
Hostname information from SLS has been added to sat status output.
Support for JSON-formatted output has been added to commands which currently
+support the --format option, such as hwinv, status, and showrev.
Many usability improvements have been made to multiple sat commands,
+mostly related to filtering command output. The following are some highlights:
--fields option to display only specific fields for subcommands which
+display tabular reports.--filter queries
+so that the first match is used, similar to --sort-by.--filter, --fields, and --reverse for summaries
+displayed by sat hwinv.sat hwinv.The default log level for stderr has been changed from “WARNING” to “INFO”. For
+more information, see SAT Logging.
With the command-line options --loglevel-stderr and --loglevel-file, the log level
+can now be configured separately for stderr and the log file.
The existing --loglevel option is now an alias for the --loglevel-stderr option.
The Podman wrapper script is the script installed at /usr/bin/sat on the
+master management NCNs by the cray-sat-podman RPM that runs the cray-sat
+container in podman. The following subsections detail improvements that were
+made to the wrapper script in this release.
cray-sat ContainerThe Podman wrapper script that launches the cray-sat container with podman
+has been modified to mount the user’s current directory and home directory into
+the cray-sat container to provide access to local files in the container.
The man page for the Podman wrapper script, which is accessed by typing man sat on a master management NCN, has been improved to document the following:
Fixed issues with redirecting stdout and stderr, and piping output to commands,
+such as awk, less, and more.
A new sat option has been added to configure the HTTP timeout length for
+requests to the API gateway. For more information, refer to sat-man sat.
sat bootsys ImprovementsMany improvements and fixes have been made to sat bootsys. The following are some
+highlights:
--excluded-ncns option, which can be used to omit NCNs
+from the platform-services and ncn-power stages in case they are
+inaccessible.sat bootsys shutdown now prompt the user to
+continue before proceeding. A new option, --disruptive, will bypass this.platform-services
+stage of sat bootsys boot.sat xname2nid Improvementssat xname2nid can now recursively expand slot, chassis, and cabinet XNames to
+a list of NIDs in those locations.
A new --format option has been added to sat xname2nid. It sets the output format to
+either “range” (the default) or “NID”. The “range” format displays NIDs in a
+compressed range format suitable for use with a workload manager like Slurm.
v2 HSM APIThe commands which interact with HSM (for example, sat status and sat hwinv) now
+use the v2 HSM API.
sat diag Limited to HSN Switchessat diag will now only operate against HSN switches by default. These are the
+only controllers that support running diagnostics with HMJTD.
sat showrev EnhancementsA column has been added to the output of sat showrev that indicates whether a
+product version is “active”. The definition of “active” varies across products,
+and not all products may set an “active” version.
For SAT, the active version is the one with its hosted-type package repository in
+Nexus set as the member of the group-type package repository in Nexus,
+meaning that it will be used when installing the cray-sat-podman RPM.
cray-sat Container Image Size ReductionThe size of the cray-sat container image has been approximately cut in half by
+leveraging multi-stage builds. This also improved the repeatability of the unit
+tests by running them in the container.
Minor bug fixes were made in cray-sat and in cray-sat-podman. For full change lists,
+refer to each repository’s CHANGELOG.md file.
We released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named ncn-personalization.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-Diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+programs on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and config file options +have been removed. For more information, see Remove Obsolete Configuration +File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes masters and workers.
sat firmware.cray-sat container image.sat firmware command.We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes management cluster (workers and
+masters). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Config File Location ChangeThe default location of the SAT config file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own config files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment variable.
Additionally, if a config file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies option.--list-node-accels option. The
+count of node accelerators is also included for each node.--list-node-accel-risers
+option. The count of node accelerator risers is also included for each node.--list-node-hsn-nics
+option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For more +information, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding config-file
+option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding config file options were
+deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchFor more information on each of these commands, see the System Admin Toolkit Command +Overview and the table +of commands in the SAT Authentication section +of this document.
+ + + + + +By default, SAT uses Boot Orchestration Service (BOS) version one. You can
+select the BOS version to use for individual commands with the --bos-version
+option. For more information on this option, refer to the man page for a specific
+command.
You can also configure the BOS version to use in the SAT config file. Do this
+under the api_version setting in the bos section of the config file. If
+the system is using an existing SAT config file from an older version of SAT,
+the bos section might not exist. In that case, add the bos section with the
+BOS version desired in the api_version setting.
Find the SAT config file at ~/.config/sat/sat.toml, and look for a section
+like this:
[bos]
+api_version = "v1"
+In this example, SAT is using BOS version "v1".
Change the line specifying the api_version to the BOS version desired (for
+example, "v2").
[bos]
+api_version = "v2"
+If applicable, uncomment the api_version line.
If the system is using an existing SAT config file from a recent version of
+SAT, the api_version line might be commented out like this:
[bos]
+# api_version = "v2"
+If the line is commented out, SAT will still use the default BOS
+version. To ensure a different BOS version is used, uncomment the
+api_version line by removing # at the beginning of the line.
SAT provides an automated solution for creating CFS configurations, building +and configuring images in IMS, and creating BOS session templates based on a +given input file which defines how those configurations, images, and session +templates should be created.
+This automated process centers around the sat bootprep command. Man page
+documentation for sat bootprep can be viewed similarly to other SAT commands.
ncn-m001# sat-man sat-bootprep
+sat bootprep is used to create CFS configurations, build and
+rename IMS images, and create BOS session templates which tie the
+configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
+including (but not limited to) performing BOS operations (such as creating BOS
+sessions), powering on and off cabinets, and checking the state of the system
+prior to shutdown.
The input file provided to sat bootprep is a YAML-formatted file containing
+information which CFS, IMS, and BOS use to create configurations, images, and
+BOS session templates respectively. Writing and modifying these input files is
+the main task associated with using sat bootprep. An input file is composed of
+three main sections, one each for configurations, images, and session templates.
+These sections may be specified in any order, and any of the sections may be
+omitted if desired.
The sat bootprep input file is validated against a versioned schema
+definition. The input file should specify the version of the schema with which
+it is compatible under a schema_version key. For example:
---
+schema_version: 1.0.2
+The current sat bootprep input file schema version can be viewed with the
+following command:
ncn-m001# sat bootprep view-schema | grep '^version:'
+version: '1.0.2'
+The sat bootprep run command validates the schema version specified
+in the input file. The command also makes sure that the schema version
+of the input file is compatible with the schema version understood by the
+current version of sat bootprep. For more information on schema version
+validation, refer to the schema_version property description in the bootprep
+input file schema. For more information on viewing the bootprep input file
+schema in either raw form or user-friendly HTML form, see Viewing the Exact
+Schema Specification or
+Generating User-Friendly Documentation.
The default sat bootprep input files provided by the hpc-csm-software-recipe
+release distribution already contain the correct schema version.
The CFS configurations are defined under a configurations key. Under this
+key, you can list one or more configurations to create. For each
+configuration, give a name in addition to the list of layers that
+comprise the configuration.
Each layer can be defined by a product name and optionally a version number, +commit hash, or branch in the product’s configuration repository. If this +method is used, the layer is created in CFS by looking up relevant configuration +information (including the configuration repository and commit information) from +the cray-product-catalog Kubernetes ConfigMap as necessary. A version may be +supplied. However, if it is absent, the version is assumed to be the latest +version found in the cray-product-catalog.
+Alternatively, a configuration layer can be defined by explicitly referencing
+the desired configuration repository. You must then specify the intended version
+of the Ansible playbooks by providing a branch name or commit hash with branch
+or commit.
The following example shows a CFS configuration with two layers. The first +layer is defined in terms of a product name and version, and the second layer +is defined in terms of a Git clone URL and branch:
+---
+configurations:
+- name: example-configuration
+ layers:
+ - name: example-product
+ playbook: example.yml
+ product:
+ name: example
+ version: 1.2.3
+ - name: another-example-product
+ playbook: another-example.yml
+ git:
+ url: "https://vcs.local/vcs/another-example-config-management.git"
+ branch: main
+When sat bootprep is run against an input file, a CFS configuration is created
+corresponding to each configuration in the configurations section. For
+example, the configuration created from an input file with the layers listed
+above might look something like the following:
{
+ "lastUpdated": "2022-02-07T21:47:49Z",
+ "layers": [
+ {
+ "cloneUrl": "https://vcs.local/vcs/example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "example product",
+ "playbook": "example.yml"
+ },
+ {
+ "cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "another example product",
+ "playbook": "another-example.yml"
+ }
+ ],
+ "name": "example-configuration"
+}
+The IMS images are defined under an images key. Under the images key, the
+user may define one or more images to be created in a list. Each element of the
+list defines a separate IMS image to be built and/or configured. Images must
+contain a name key and a base key.
The name key defines the name of the resulting IMS image. The base key
+defines the base image to be configured or the base recipe to be built and
+optionally configured. One of the following keys must be present under the
+base key:
ims key to specify an existing image or recipe in IMS.product key to specify an image or recipe provided by a
+particular version of a product. Note that this is only possible if the
+product provides a single image or recipe.image_ref key to specify another image from the input file
+using its ref_name.Images may also contain the following keys:
+configuration key to specify a CFS configuration with which to
+customize the built image. If a configuration is specified, then configuration
+groups must also be specified using the configuration_group_names key.ref_name key to specify a unique name that can refer to this image
+within the input file in other images or in session templates. The ref_name
+key allows references to images from the input file that have dynamically
+generated names as described in
+Dynamic Variable Substitutions.description key to describe the image in the bootprep input file.
+Note that this key is not currently used.Here is an example of an image using an existing IMS recipe as its base. This
+example builds an IMS image from that recipe. It then configures it with
+a CFS configuration named example-compute-config. The example-compute-config
+CFS configuration can be defined under the configurations key in the same
+input file, or it can be an existing CFS configuration. Running sat bootprep
+against this input file results in an image named example-compute-image.
images:
+- name: example-compute-image
+ description: >
+ An example compute node image built from an existing IMS recipe.
+ base:
+ ims:
+ name: example-compute-image-recipe
+ type: recipe
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+Here is an example showing the definition of two images. The first image is
+built from a recipe provided by the cos product. The second image uses the
+first image as a base and configures it with a configuration named
+example-compute-config. The value of the first image’s ref_name key is used
+in the second image’s base.image_ref key to specify it as a dependency.
+Running sat bootprep against this input file results in two images, the
+first named example-cos-image and the second named example-compute-image.
images:
+- name: example-cos-image
+ ref_name: example-cos-image
+ description: >
+ An example image built from a recipe provided by the COS product.
+ base:
+ product:
+ name: cos
+ version: 2.3.101
+ type: recipe
+- name: example-compute-image
+ description: >
+ An example image built from a recipe provided by the COS product.
+ base:
+ image_ref: example-cos-image
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+The BOS session templates are defined under the session_templates key. Each
+session template must provide values for the name, image, configuration,
+and bos_parameters keys. The name key defines the name of the resulting BOS
+session template. The image key defines the image to use in the BOS session
+template. One of the following keys must be present under the image key:
ims key to specify an existing image or recipe in IMS.image_ref key to specify another image from the input file
+using its ref_name.The configuration key defines the CFS configuration specified
+in the BOS session template.
The bos_parameters key defines parameters that are passed through directly to
+the BOS session template. The bos_parameters key should contain a boot_sets
+key, and each boot set in the session template should be specified under
+boot_sets. Each boot set can contain the following keys, all of
+which are optional:
kernel_parameters key to specify the parameters passed to the kernel on the command line.network key to specify the network over which the nodes boot.node_list key to specify the nodes to add to the boot set.node_roles_groups key to specify the HSM roles to add to the boot set.node_groups key to specify the HSM groups to add to the boot set.rootfs_provider key to specify the root file system provider.rootfs_provider_passthrough key to specify the parameters to add to the rootfs=
+kernel parameter.As mentioned above, the parameters under bos_parameters are passed through
+directly to BOS. For more information on the properties of a BOS boot set,
+refer to BOS Session Templates in the Cray
+System Management Documentation.
Here is an example of a BOS session template that refers to an existing IMS +image by name:
+session_templates:
+- name: example-session-template
+ image:
+ ims:
+ name: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+Here is an example of a BOS session template that refers to an image from the
+input file by its ref_name. This requires that an image defined in the input
+file specifies example-image as the value of its ref_name key.
session_templates:
+- name: example-session-template
+ image:
+ image_ref: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+The HPC CSM Software Recipe provides a manifest defining the versions of each
+HPC software product included in the recipe. These product versions can be used
+in the sat bootprep input file with Jinja2 template syntax.
By default, the sat bootprep command uses the product versions from the
+latest installed version of the HPC CSM Software Recipe. However, you can
+override this with the --recipe-version command line argument to sat bootprep run.
For example, to explicitly select the 22.11.0 version of the HPC CSM Software
+Recipe, specify --recipe-version 22.11.0:
ncn-m001# sat bootprep run --recipe-version 22.11.0 compute-and-uan-bootprep.yaml
+The entire sat bootprep input file is not rendered by the Jinja2 template
+engine. Jinja2 template rendering of the input file is performed individually
+for each supported value. The values of the following keys support rendering as
+a Jinja2 template:
name key of each configuration under the configurations key.layers key in a
+configuration:
+namegit.branchproduct.versionproduct.branchimages key:
+namebase.product.versionconfigurationsession_templates key:
+nameconfigurationYou can use Jinja2 built-in filters in values of any of the keys listed above. +In addition, Python string methods can be called on the string variables.
+HPC CSM Software Recipe variables are available, and you can use them in the values
+of the keys listed above. View these variables by cloning the hpc-csm-software-recipe
+repository from VCS and accessing the product_vars.yaml file on the branch that
+corresponds to the targeted version of the HPC CSM Software Recipe.
Set up a shell script to access the password for the crayvcs user:
ncn-m001# cat > vcs-creds-helper.sh <<EOF
+#!/bin/bash
+kubectl get secret -n services vcs-user-credentials -o jsonpath={.data.vcs_password} | base64 -d
+EOF
+Ensure vcs-creds-helper.sh is executable:
ncn-m001# chmod u+x vcs-creds-helper.sh
+Set the GIT_ASKPASS environment variable to the path to the
+vcs-creds-helper.sh script:
ncn-m001# export GIT_ASKPASS="$PWD/vcs-creds-helper.sh"
+Clone the hpc-csm-software-recipe repository:
ncn-m001# git clone https://crayvcs@api-gw-service-nmn.local/vcs/cray/hpc-csm-software-recipe.git
+Change the directory to the hpc-csm-software-recipe repository:
ncn-m001# cd hpc-csm-software-recipe
+View the versions of the HPC CSM Software Recipe on the system:
+ncn-m001# git branch -r
+Check out the branch of the hpc-csm-software-recipe repository that corresponds to
+the targeted HPC CSM Software Recipe version. For example, for recipe version
+22.11.0:
ncn-m001# git checkout cray/hpc-csm-software-recipe/22.11.0
+View the contents of the file product_vars.yaml in the clone of the
+repository:
ncn-m001# cat product_vars.yaml
+The variables defined in the product_vars.yaml file can be used in the values
+that support Jinja2 templates. A variable is specified by a dot-separated path,
+with each component of the path representing a key in the YAML file. For
+example, a version of the COS product appears as follows in the
+product_vars.yaml file:
cos:
+ version: 2.4.76
+This COS version can be used by specifying cos.version within a value in the
+input file.
The following example bootprep input file shows how a COS version can be +used in a bootprep input file that creates a CFS configuration for computes. +Only one layer is shown for brevity.
+---
+configurations:
+- name: compute-{{recipe.version}}
+ layers:
+ - name: cos-compute-integration-{{cos.version}}
+ playbook: cos-compute.yaml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration-{{cos.version}}
+Note: When the value of a key in the bootprep input file is a Jinja2 +expression, it must be quoted to pass YAML syntax checking.
+Jinja2 expressions can also use filters and Python’s built-in string methods to
+manipulate the variable values. For example, suppose only the major and minor
+components of a COS version are to be used in the branch name for the COS
+layer of the CFS configuration. You can use the split string method to
+achieve this as follows:
---
+configurations:
+- name: compute-{{recipe.version}}
+ layers:
+ - name: cos-compute-integration-{{cos.version}}
+ playbook: cos-compute.yaml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration-{{cos.version.split('.')[0]}}-{{cos.version.split('.')[1]}}
+Additional variables are available besides the product version variables +provided by the HPC CSM Software Recipe. (For more information, see HPC +CSM Software Recipe Variable Substitutions.) +These additional variables are dynamic because their values are +determined at run-time based on the context in which they appear. Available +dynamic variables include the following:
+base.name can be used in the name of an image under the
+images key. The value of this variable is the name of the IMS image or
+recipe used as the base of this image.image.name can be used in the name of a session template
+under the session_templates key. The value of this variable is the name of
+the IMS image used in this session template.These variables reduce the need to duplicate values throughout the sat bootprep input file and make the following use cases possible:
This section provides an example bootprep input file. It also gives +instructions for obtaining the default bootprep input files delivered +with a release of the HPC CSM Software Recipe.
+The following bootprep input file provides an example of using most of the +features described in previous sections. It is not intended to be a complete +bootprep file for the entire CSM product.
+---
+configurations:
+- name: compute-{{recipe.version}}
+ layers:
+ - name: cos-compute-integration-{{cos.version}}
+ playbook: site.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration-{{cos.version}}
+ - name: cpe-pe_deploy-integration-{{cpe.version}}
+ playbook: pe_deploy.yml
+ product:
+ name: cpe
+ version: "{{cpe.version}}"
+ branch: integration-{{cpe.version}}
+
+images:
+- name: "{{base.name}}"
+ ref_name: base_cos_image
+ base:
+ product:
+ name: cos
+ type: recipe
+ version: "{{cos.version}}"
+
+- name: compute-{{base.name}}
+ ref_name: compute_image
+ base:
+ image_ref: base_cos_image
+ configuration: compute-{{recipe.version}}
+ configuration_group_names:
+ - Compute
+
+session_templates:
+- name: compute-{{recipe.version}}
+ image:
+ image_ref: compute_image
+ configuration: compute-{{recipe.version}}
+ bos_parameters:
+ boot_sets:
+ compute:
+ kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
+ node_roles_groups:
+ - Compute
+ rootfs_provider_passthrough: "dvs:api-gw-service-nmn.local:300:hsn0,nmn0:0"
+Default bootprep input files are delivered by the HPC CSM Software Recipe
+product. You can access these files by cloning the hpc-csm-software-recipe
+repository.
To do this, follow steps 1-7 of the procedure in Viewing HPC CSM Software Recipe
+Variables. Then, access the files in the
+bootprep directory of that repository:
ncn-m001# ls bootprep/
+The sat bootprep generate-example command was not updated for
+recent bootprep schema changes. It is recommended that you instead use the
+default bootprep input files described in Accessing Default Bootprep Input
+Files. The sat bootprep generate-example command will be updated in a future release of SAT.
You might need to edit the default bootprep input files delivered by the HPC +CSM Software Recipe for your system. Here are some examples of how to edit +the files.
+Before running sat bootprep, HPE recommends reading the bootprep input files
+and paying specific attention to the branch parameters. Some HPE Cray EX
+products require system-specific changes on a working branch of VCS. For these
+products, the default bootprep input files assume certain naming conventions for
+the VCS branches. The files refer to a particular branch of a product’s
+configuration management repository.
Thus, it is important to confirm that the bootprep input files delivered by the
+HPC CSM Software Recipe match the actual system branch names. For example, the
+COS product’s CFS configuration layer is defined as follows in the default
+management-bootprep.yaml bootprep input file.
- name: cos-ncn-integration-{{cos.version}}
+ playbook: ncn.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration-{{cos.version}}
+The default file is assuming that system-specific Ansible configuration changes
+for the COS product in VCS are stored in a branch named
+integration-{{cos.version}}. If the version being installed is COS 2.4.99,
+sat bootprep looks for a branch named integration-2.4.99 from which to
+create CFS configuration layers.
You can create VCS working branches that are not the default bootprep input file
+branch names. A simple example of this is using cne-install to update working
+VCS branches. If you use cne-install to update working VCS branches, (namely in
+the update_working_branches stage), you create or update the branches specified
+by the -B WORKING_BRANCH command line option. For example, consider the
+following cne-install command.
ncn-m001# ./cne-install install \
+ -B integration \
+ -s deploy_products \
+ -e update_working_branches
+Products installed with this cne-install example use the working branch
+integration for system-specific changes to VCS. The branch specified by the
+-B option must match the branch specified in the bootprep input file.
In another example, to use the branch integration for COS instead of
+integration-{{cos.version}}, edit the bootprep input file so it reads as
+follows.
- name: cos-ncn-integration-{{cos.version}}
+ playbook: ncn.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration
+The default bootprep input file for management CFS configurations
+(management-bootprep.yaml) creates configurations that have names specified
+within the input file. For example, in the bootprep input files included in the
+22.11 HPC CSM Software Recipe, the following configurations are named:
ncn-personalizationncn-image-customizationThese default management CFS configuration names might be acceptable for your
+system. However, it is possible to create other names. sat bootprep creates
+whatever configurations are specified in the input file. For example, to
+create a NCN node personalization configuration named
+ncn-personalization-test, edit the file as follows.
configurations:
+- name: ncn-personalization-test
+ layers:
+ ...
+For management configurations, use sat status to identify the current
+desired configuration for each of the management nodes.
ncn-m001# sat status --fields xname,role,subrole,desiredconfig --filter role=management
++----------------+------------+---------+---------------------+
+| xname | Role | SubRole | Desired Config |
++----------------+------------+---------+---------------------+
+| x3000c0s1b0n0 | Management | Master | ncn-personalization |
+| x3000c0s3b0n0 | Management | Master | ncn-personalization |
+| x3000c0s5b0n0 | Management | Master | ncn-personalization |
+| x3000c0s7b0n0 | Management | Worker | ncn-personalization |
+| x3000c0s9b0n0 | Management | Worker | ncn-personalization |
+| x3000c0s11b0n0 | Management | Worker | ncn-personalization |
+| x3000c0s13b0n0 | Management | Worker | ncn-personalization |
+| x3000c0s17b0n0 | Management | Storage | ncn-personalization |
+| x3000c0s19b0n0 | Management | Storage | ncn-personalization |
+| x3000c0s21b0n0 | Management | Storage | ncn-personalization |
+| x3000c0s25b0n0 | Management | Worker | ncn-personalization |
++----------------+------------+---------+---------------------+
+To overwrite the desired configuration using sat bootprep, ensure the bootprep
+input file specifies to create a configuration with the same name
+(ncn-personalization in the example above). To create a different configuration,
+ensure the bootprep input file specifies to create a configuration with a
+different name than the desired configuration (different than ncn-personalization
+in the example above).
When working with a given HPC CSM Software Recipe, it might be necessary to +upgrade a single HPE Cray EX product past the default version given in the +recipe. However, you might still want to use the other default product versions +contained in that recipe. To do this, first upgrade the single product. For +more information, refer to the upgrade instructions in that product’s +documentation.
+After the product is upgraded, you must override its default version in subsequent
+runs of sat bootprep. The following process explains how to do this. In this
+example, all the default product versions from the 22.11 software recipe are
+used except for COS. The COS default product version is overridden to version
+2.4.199 instead, and the CFS configurations in management-bootprep.yaml are
+created.
Ensure you have a local copy of the default bootprep input files.
+For more information, see Accessing Default Bootprep Input +Files.
+Edit the product_vars.yaml file to change the default product version.
ncn-m001# vim product_vars.yaml
+Confirm the new product version in the edited product_vars.yaml file.
ncn-m001# grep -A1 cos: `product_vars.yaml`:
+cos:
+ version: 2.4.199
+Use the --vars-file option when running sat bootprep to override the
+default product version.
You must run this command from the directory containing the product_vars.yaml
+file. The product_vars.yaml file must also be specified when using the
+--vars-file option. It is not sufficient to just edit the file.
ncn-m001# sat bootprep run --vars-file product_vars.yaml bootprep/management-bootprep.yaml
+Note: This example is specific to creating the configurations defined in
+management-bootprep.yaml. Review what configurations, images, or session templates
+you intend to create by viewing the input file.
The contents of the YAML input files described above must conform to a schema +which defines the structure of the data. The schema definition is written using +the JSON Schema format. (Although the format is named “JSON Schema”, the schema +itself is written in YAML as well.) More information, including introductory +materials and a formal specification of the JSON Schema metaschema, can be found +on the JSON Schema website.
+To view the exact schema specification, run sat bootprep view-schema.
ncn-m001# sat bootprep view-schema
+---
+$schema: "https://json-schema.org/draft/2020-12/schema"
+...
+title: Bootprep Input File
+description: >
+ A description of the set of CFS configurations to create, the set of IMS
+ images to create and optionally customize with the defined CFS configurations,
+ and the set of BOS session templates to create that reference the defined
+ images and configurations.
+type: object
+additionalProperties: false
+properties:
+ ...
+The raw schema definition can be difficult to understand without experience +working with JSON Schema specifications. For this reason, a feature is included +that generates user-friendly HTML documentation for the input file schema. This +HTML documentation can be browsed with your preferred web browser.
+Create a documentation tarball using sat bootprep.
ncn-m001# sat bootprep generate-docs
+INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
+An alternate output directory can be specified with the --output-dir
+option. The generated tarball is always named bootprep-schema-docs.tar.gz.
ncn-m001# sat bootprep generate-docs --output-dir /tmp
+INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
+From another machine, copy the tarball to a local directory.
+another-machine$ scp root@ncn-m001:bootprep-schema-docs.tar.gz .
+Extract the contents of the tarball and open the contained index.html.
another-machine$ tar xzvf bootprep-schema-docs.tar.gz
+x bootprep-schema-docs/
+x bootprep-schema-docs/index.html
+x bootprep-schema-docs/schema_doc.css
+x bootprep-schema-docs/schema_doc.min.js
+another-machine$ open bootprep-schema-docs/index.html
+The SAT Grafana Dashboards display messages that are generated by the HSN (High Speed Network) and reported through +Redfish. The messages are displayed based on severity.
+Grafana can be accessed via web browser at the following URL:
+https://sma-grafana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Grafana being https://sma-grafana.cmn.EXAMPLE_DOMAIN.com
For more information on accessing the Grafana Dashboards, refer to Access the Grafana Monitoring UI in the +SMA product documentation.
+For more information on the interpretation of metrics for the SAT Grafana Dashboards, refer to “Fabric Telemetry +Kafka Topics” in the SMA product documentation.
+There are four Fabric Telemetry dashboards used in SAT that report on the HSN. Two contain chart panels and two display +telemetry in a tabular format.
+| Dashboard Name | +Display Type | +
|---|---|
| Fabric Congestion | +Chart Panels | +
| Fabric RFC3635 | +Chart Panels | +
| Fabric Errors | +Tabular Format | +
| Fabric Port State | +Tabular Format | +
The tabular format presents a single point of telemetry for a given location and metric, either because the telemetry +is not numerical or that it changes infrequently. The value shown is the most recently reported value for that location +during the time range selected, if any. The interval setting is not used for tabular dashboards.
+Shows the Interval and Locations Options for the available telemetry.
+
The value of the Interval option sets the time resolution of the received telemetry. This works a bit like a +histogram, with the available telemetry in an interval of time going into a “bucket” and averaging out to a single +point on the chart or table. The special value auto will choose an interval based on the time range selected.
+For more information, refer to Grafana Templates and Variables.
+The Locations option allows restriction of the telemetry shown by locations, either individual links or all links +in a switch. The selection presented updates dynamically according to time range, except for the errors dashboard, +which always has entries for all links and switches, although the errors shown are restricted to the selected time +range.
+The chart panels for the RFC3635 and Congestion dashboards allow selection of a single location from the chart’s legend +or the trace on the chart.
+
SAT Grafana Dashboards provide system administrators a way to view fabric telemetry data across all Rosetta switches in +the system and assess the past and present health of the high-speed network. It also allows the ability to drill down +to view data for specific ports on specific switches.
+This dashboard contains the variable, Port Type not found in the other dashboards. The possible values are edge, +local, and global and correspond to the link’s relationship to the network topology. The locations presented in the +panels are restricted to the values (any combination, defaults to “all”) selected.
+The metric values for links of a given port type are similar in value to each other but very distinct from the values of +other types. If the values for different port types are all plotted together, the values for links with lower values are +indistinguishable from zero when plotted.
+The port type of a link is reported as a port state “subtype” event when defined at port initialization.
+
This dashboard reports error counters in a tabular format in three panels.
+There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+Unlike other dashboards, the locations presented are all locations in the system rather than having telemetry within +the time range selected. However, the values are taken from telemetry within the time range.
+
There is no Interval option because this parameter is not used to set a coarseness of the data. Only a single value +is presented that displays the most recent value in the time range.
+The Fabric Port State telemetry is distinct because it typically is not numeric. It also updates infrequently, so a +long time range may be necessary to obtain any values. Port State is refreshed daily, so a time range of 24 hours +results in all states for all links in the system being shown.
+The three columns named, group, switch, and port are not port state events, but extra information included with +all port state events.
+
For more information on performance counters, refer to +Definitions of Managed Objects for the Ethernet-like Interface Types, +an Internet standards document.
+Because these metrics are counters that only increase over time, the values plotted are the change in the counter’s +value over the interval setting.
+ + + + + +Kibana is an open source analytics and visualization platform designed to search, view, and interact with data stored +in Elasticsearch indices. Kibana runs as a web service and has a browser-based interface. It offers visual output of +node data in the forms of charts, tables and maps that display real-time Elasticsearch queries. Viewing system data in +this way breaks down the complexity of large data volumes into easily understood information.
+Kibana can be accessed via web browser at the following URL:
+https://sma-kibana.cmn.<site-domain>The value of site-domain can be obtained as follows:
ncn-m001:~ # kubectl get secret site-init -n loftsman -o jsonpath='{.data.customizations\.yaml}' | \
+ base64 -d | grep "external:"
+That command will produce the following output, for example:
+ external: EXAMPLE_DOMAIN.com
+This would result in the address for Kibana being https://sma-kibana.cmn.EXAMPLE_DOMAIN.com
For more information on accessing the Kibana Dashboards, refer to View Logs Via Kibana in the SMA product +documentation.
+Additional details about the AER, ATOM, Heartbeat, Kernel, MCE, and RAS Daemon Kibana Dashboards are included in this +table.
+| Dashboard | +Short Description | +Long Description | +Kibana Visualization and Search Name | +
|---|---|---|---|
sat-aer |
+AER corrected | +Corrected Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-corrected Search: sat-aer-corrected |
+
sat-aer |
+AER fatal | +Fatal Advanced Error Reporting messages from PCI Express devices on each node. | +Visualization: aer-fatal Search: sat-aer-fatal |
+
sat-atom |
+ATOM failures | +Application Task Orchestration and Management tests are run on a node when a job finishes. Test failures are logged. | +sat-atom-failed |
+
sat-atom |
+ATOM admindown |
+Application Task Orchestration and Management test failures can result in nodes being marked admindown. An admindown node is not available for job launch. |
+sat-atom-admindown |
+
sat-heartbeat |
+Heartbeat loss events | +Heartbeat loss event messages reported by the hbtd pods that monitor for heartbeats across nodes in the system. |
+sat-heartbeat |
+
sat-kernel |
+Kernel assertions | +The kernel software performs a failed assertion when some condition represents a serious fault. The node goes down. | +sat-kassertions |
+
sat-kernel |
+Kernel panics | +The kernel panics when something is seriously wrong. The node goes down. | +sat-kernel-panic |
+
sat-kernel |
+Lustre bugs (LBUGs) | +The Lustre software in the kernel stack performs a failed assertion when some condition related to file system logic represents a serious fault. The node goes down. | +sat-lbug |
+
sat-kernel |
+CPU stalls | +CPU stalls are serous conditions that can reduce node performance, and sometimes cause a node to go down. Technically these are Read-Copy-Update stalls where software in the kernel stack holds onto memory for too long. Read-Copy-Update is a vital aspect of kernel performance and rather esoteric. | +sat-cpu-stall |
+
sat-kernel |
+Out of memory | +An Out Of Memory (OOM) condition has occurred. The kernel must kill a process to continue. The kernel will select an expendable process when possible. If there is no expendable process the node usually goes down in some manner. Even if there are expendable processes the job is likely to be impacted. OOM conditions are best avoided. | +sat-oom |
+
sat-mce |
+MCE | +Machine Check Exceptions (MCE) are errors detected at the processor level. | +sat-mce |
+
sat-rasdaemon |
+rasdaemon errors |
+Errors from the rasdaemon service on nodes. The rasdaemon service is the Reliability, Availability, and Serviceability Daemon, and it is intended to collect all hardware error events reported by the Linux kernel, including PCI and MCE errors. This may include certain HSN errors in the future. |
+sat-rasdaemon-error |
+
sat-rasdaemon |
+rasdaemon messages |
+All messages from the rasdaemon service on nodes. |
+sat-rasdaemon |
+
By default, search highlighting is enabled. This procedure instructs how to disable search highlighting.
+The Kibana Dashboard should be open on your system.
+Navigate to Management
+Navigate to Advanced Settings in the Kibana section, below the Elastic search section
+Scroll down to the Discover section
+Change Highlight results from on to off
+Click Save to save changes
+The AER Dashboard displays errors that come from the PCI Express Advanced Error Reporting (AER) driver. These errors +are split up into separate visualizations depending on whether they are fatal or corrected errors.
+Go to the dashboard section.
+Select sat-aer dashboard.
Choose the time range of interest.
+View the Corrected and Fatal Advanced Error Reporting messages from PCI Express devices on each node. View the +matching log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on +the left. If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass +next to each NID.
+The ATOM (Application Task Orchestration and Management) Dashboard displays node failures that occur during health
+checks and application test failures. Some test failures are of possible interest even though a node is not marked
+admindown or otherwise fails. They are of clear interest if a node is marked admindown, and might provide
+clues if a node otherwise fails. They might also show application problems.
HPE Cray EX is installed on the system along with the System Admin Toolkit, which contains the ATOM Kibana Dashboard.
+Go to the dashboard section.
+Select sat-atom dashboard.
Choose the time range of interest.
+View any nodes marked admindown and any ATOM test failures. These failures occur during health checks and
+application test failures. Test failures marked admindown are important to note. View the matching log messages
+in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. If desired,
+results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to each NID.
The Heartbeat Dashboard displays heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd
+pods are responsible for monitoring nodes in the system for heartbeat loss.
Go to the dashboard section.
+Select sat-heartbeat dashboard.
Choose the time range of interest.
+View the heartbeat loss messages that are logged by the hbtd pods in the system. The hbtd pods are responsible
+for monitoring nodes in the system for heartbeat loss. View the matching log messages in the panel.
The Kernel Dashboard displays compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. +The messages reveal if Lustre has experienced a fatal error on any compute nodes in the system. A CPU stall is a serious +problem that might result in a node failure. Out-of-memory conditions can be due to applications or system problems and +may require expert analysis. They provide useful clues for some node failures and may reveal if an application is using +too much memory.
+Go to the dashboard section.
+Select sat-kernel dashboard.
Choose the time range of interest.
+View the compute node failures such as kernel assertions, kernel panics, and Lustre LBUG messages. View the matching +log messages in the panel(s) on the right, and view the counts of each message per NID in the panel(s) on the left. +If desired, results can be filtered by NID by clicking the icon showing a + inside a magnifying glass next to +each NID.
+The MCE Dashboard displays CPU detected processor-level hardware errors.
+Go to the dashboard section.
+Select sat-mce dashboard.
Choose the time range of interest.
+View the Machine Check Exceptions (MCEs) listed including the counts per NID (node). For an MCE, the CPU number and +DIMM number can be found in the message, if applicable. View the matching log messages in the panel(s) on the right, +and view the counts of each message per NID in the panel(s) on the left. If desired, results can be filtered by NID +by clicking the icon showing a + inside a magnifying glass next to each NID.
+The RAS Daemon Dashboard displays errors that come from the Reliability, Availability, and Serviceability (RAS) daemon
+service on nodes in the system. This service collects all hardware error events reported by the Linux kernel, including
+PCI and MCE errors. As a result there may be some duplication between the messages presented here and the messages
+presented in the MCE and AER dashboards. This dashboard splits up the messages into two separate visualizations, one
+for only messages of severity emerg or err and another for all messages from rasdaemon.
Go to the dashboard section.
+Select sat-rasdaemon dashboard.
Choose the time range of interest.
+View the errors that come from the Reliability, Availability, and Serviceability (RAS) daemon service on nodes in +the system. View the matching log messages in the panel(s) on the right, and view the counts of each message per NID +in the panel(s) on the left. If desired, results can be filtered by NID by clicking the icon showing a + inside +a magnifying glass next to each NID.
+SAT can optionally be installed and configured on an external system to interact +with CSM over the CAN.
+Most SAT subcommands work by accessing APIs which are reachable via the CAN. +However, certain SAT commands depend on host-based functionality on the +management NCNs and will not work from an external system. This includes the +following:
+platform-services and ncn-power stages of sat bootsys--local option of sat showrevInstalling SAT on an external system is not an officially supported configuration. +These instructions are provided “as-is” with the hope that they can useful for +users who desire additional flexibility.
+Certain additional steps may need to be taken to install and configure SAT +depending on the configuration of the external system in use. These additional +steps may include provisioning virtual machines, installing packages, or +configuring TLS certificates, and these steps are outside the scope of this +documentation. This section covers only the steps needed to configure SAT to +use externally-accessible API endpoints exposed by CSM.
+kubectl, openssh, git, and curl are installed on the external system.Create a Python virtual environment.
+$ SAT_VENV_PATH="$(pwd)/venv"
+$ python3 -m venv ${SAT_VENV_PATH}
+$ . ${SAT_VENV_PATH}/bin/activate
+Clone the SAT source code.
+To use SAT version 3.21, this example clones the release/3.21 branch of
+Cray-HPE/sat.
(venv) $ git clone --branch=release/3.21 https://github.com/Cray-HPE/sat.git
+Set up the SAT CSM Python dependencies to be installed from their source code.
+SAT CSM Python dependency packages are not currently distributed publicly as
+source packages or binary distributions. They must be installed from
+their source code hosted on GitHub. Also, to install the cray-product-catalog
+Python package, you must first clone it locally. Use the following steps to
+modify the SAT CSM Python dependencies so they can be installed from their source
+code.
Clone the source code for cray-product-catalog.
(venv) $ git clone --branch v1.6.0 https://github.com/Cray-HPE/cray-product-catalog
+In the cray-product-catalog directory, create a file named .version
+that contains the version of cray-product-catalog.
(venv) $ echo 1.6.0 > cray-product-catalog/.version
+Open the “locked” requirements file in a text editor.
+(venv) $ vim sat/requirements.lock.txt
+Update the line containing cray-product-catalog so that it reflects the
+local path to cray-product-catalog.
It should read as follows.
+./cray-product-catalog
+For versions of SAT newer than 3.19, change the line containing csm-api-client
+to read as follows.
csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+(Optional) Confirm that requirements.lock.txt is modified as expected.
Note: For versions newer than 3.19, you will see both cray-product-catalog
+and csm-api-client. For version 3.19 and older, you will only see
+cray-product-catalog.
(venv) $ grep -E 'cray-product-catalog|csm-api-client' sat/requirements.lock.txt
+./cray-product-catalog
+csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+Install the modified SAT dependencies.
+(venv) $ pip install -r sat/requirements.lock.txt
+...
+Install the SAT Python package.
+(venv) $ pip install ./sat
+...
+(Optional) Add the sat virtual environment to the user’s PATH environment
+variable.
If a shell other than bash is in use, replace ~/.bash_profile with the
+appropriate profile path.
If the virtual environment is not added to the user’s PATH environment
+variable, then source ${SAT_VENV_PATH}/bin/activate will need to be run before
+running any SAT commands.
(venv) $ deactivate
+$ echo export PATH=\"${SAT_VENV_PATH}/bin:${PATH}\" >> ~/.bash_profile
+$ source ~/.bash_profile
+Copy the file /etc/kubernetes/admin.conf from ncn-m001 to ~/.kube/config
+on the external system.
Note that this file contains credentials to authenticate against the Kubernetes +API as the administrative user, so it should be treated as sensitive.
+$ mkdir -p ~/.kube
+$ scp ncn-m001:/etc/kubernetes/admin.conf ~/.kube/config
+admin.conf 100% 5566 3.0MB/s 00:00
+Add a new entry for the hostname kubernetes to the external system’s
+/etc/hosts file.
The kubernetes hostname should correspond to the CAN IP address on ncn-m001.
+On CSM 1.2, this can be determined by querying the IP address of the bond0.cmn0
+interface.
$ ssh ncn-m001 ip addr show bond0.cmn0
+13: bond0.cmn0@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.11/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+$ IP_ADDRESS=10.102.1.11
+On CSM versions prior to 1.2, the CAN IP can be determined by querying the
+IP address of the vlan007 interface.
$ ssh ncn-m001 ip addr show vlan007
+13: vlan007@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.10/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+$ IP_ADDRESS=10.102.1.10
+Once the IP address is determined, add an entry to /etc/hosts mapping the
+IP address to the hostname kubernetes.
$ echo "${IP_ADDRESS} kubernetes" | sudo tee -a /etc/hosts
+10.102.1.11 kubernetes
+Modify ~/.kube/config to set the cluster server address.
The value of the server key for the kubernetes cluster under the clusters
+section should be set to https://kubernetes:6443.
---
+clusters:
+- cluster:
+ certificate-authority-data: REDACTED
+ server: https://kubernetes:6443
+ name: kubernetes
+...
+Confirm that kubectl can access the CSM Kubernetes cluster.
$ kubectl get nodes
+NAME STATUS ROLES AGE VERSION
+ncn-m001 Ready master 135d v1.19.9
+ncn-m002 Ready master 136d v1.19.9
+ncn-m003 Ready master 136d v1.19.9
+ncn-w001 Ready <none> 136d v1.19.9
+ncn-w002 Ready <none> 136d v1.19.9
+ncn-w003 Ready <none> 136d v1.19.9
+Use sat init to create a configuration file for SAT.
$ sat init
+INFO: Configuration file "/home/user/.config/sat/sat.toml" generated.
+Copy the platform CA certificates from the management NCN and configure the +certificates for use with SAT.
+If a shell other than bash is in use, replace ~/.bash_profile with the
+appropriate profile path.
$ scp ncn-m001:/etc/pki/trust/anchors/platform-ca-certs.crt .
+$ echo export REQUESTS_CA_BUNDLE=\"$(realpath platform-ca-certs.crt)\" >> ~/.bash_profile
+$ source ~/.bash_profile
+Edit the SAT configuration file to set the API and S3 hostnames.
+Externally available API endpoints are given domain names in PowerDNS, so the
+endpoints in the configuration file should each be set to
+subdomain.system-name.site-domain, where system-name and site-domain are
+replaced with the values specified during csi config init, and subdomain
+is the DNS name for the externally available service. For more information,
+refer to Externally Exposed Services in the Cray System Management
+Documentation.
The API gateway has the subdomain api, and S3 has the subdomain s3. The
+S3 endpoint runs on port 8080. The following options should be set in the
+SAT configuration file.
[api_gateway]
+host = "api.system-name.site-domain"
+
+[s3]
+endpoint = "http://s3.system-name.site-domain:8080"
+Edit the SAT configuration file to specify the Keycloak user which will be +accessing the REST API.
+[api_gateway]
+username = "user"
+Run sat auth. Enter your password when prompted.
The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For more
+information on editing Role Mappings, see Create Internal User Accounts
+in the Keycloak Shasta Realm in the
+Cray System Management Documentation.
+For more information on authentication types and authentication credentials,
+see SAT Command Authentication.
$ sat auth
+Password for user:
+Succeeded!
+Ensure the files are readable only by the current user.
+$ touch ~/.config/sat/s3_access_key \
+ ~/.config/sat/s3_secret_key
+$ chmod 600 ~/.config/sat/s3_access_key \
+ ~/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
Generate S3 credentials and write them to a local file so the SAT user can
+access S3 storage. In order to use the SAT S3 bucket, the user must generate
+the S3 access key and secret keys and write them to a local file. SAT uses
+S3 storage for several purposes, most importantly to store the site-specific
+information set with sat setrev.
$ kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ ~/.config/sat/s3_access_key
+$ kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ ~/.config/sat/s3_secret_key
+The Install and Upgrade Framework (IUF) provides commands which install, +upgrade, and deploy products on systems managed by CSM. IUF capabilities are +described in detail in the IUF +section of the +Cray System Management Documentation. +The initial install and upgrade workflows described in the +HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM +(S-8052) detail when and how to use +IUF with a new release of SAT or any other HPE Cray EX product.
+This document does not replicate install, upgrade, or deployment procedures +detailed in the Cray System Management +Documentation. This document provides +details regarding software and configuration content specific to SAT which is +needed when installing, upgrading, or deploying a SAT release. The Cray +System Management Documentation will +indicate when sections of this document should be referred to for detailed +information.
+IUF will perform the following tasks for a release of SAT.
+deliver-product stage:
+update-vcs-config stage:
+update-cfs-config stage:
+prepare-images stage:
+management-nodes-rollout stage:
+IUF uses a variety of CSM and SAT tools when performing these tasks. The IUF +section of the +Cray System Management Documentation +describes how to use these tools directly if it is desirable to use them +instead of IUF.
+This section describes SAT details that an administrator must be aware of +before running IUF stages. Entries are prefixed with Information if no +administrative action is required or Action if an administrator needs +to perform tasks outside of IUF.
+Information: This stage is only run if a VCS working branch is specified for +SAT. By default, SAT does not create or specify a VCS working branch.
+Information: This stage only applies to the management configuration and +not to the managed configuration.
+Information: This stage only applies to management images and not to +managed images.
+After installing SAT with IUF, you must complete the following SAT configuration +procedures before using SAT:
+ +...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being installed.To run SAT commands on the manager NCNs, you must first set up authentication
+to the API gateway. The admin account used to authenticate with sat auth
+must be enabled in Keycloak and must have its assigned role set to admin.
+For more information on editing Role Mappings, see Create Internal User Accounts
+in the Keycloak Shasta Realm in the Cray System Management
+Documentation. For more information on
+authentication types and authentication credentials, see SAT Command
+Authentication.
sat CLI has been installed following the IUF
+section of the
+Cray System Management Documentation.The following is the procedure to globally configure the username used by SAT and +authenticate to the API gateway.
+Generate a default SAT configuration file, if one does not exist.
+ncn-m001# sat init
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the config file already exists, it will print out the following +error.
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway
+section of the config file.
username = "crayadmin"
+Run sat auth. Enter your password when prompted.
ncn-m001# sat auth
+Password for crayadmin:
+Succeeded!
+Other sat commands are now authenticated to make requests to the API gateway.
ncn-m001# sat status
+Generate S3 credentials and write them to a local file so the SAT user can access +S3 storage. In order to use the SAT S3 bucket, the System Administrator must +generate the S3 access key and secret keys and write them to a local file. This +must be done on every Kubernetes master node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the
+site-specific information set with sat setrev (see Set System Revision
+Information).
Ensure the files are readable only by root.
ncn-m001# touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+ncn-m001# chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+Write the credentials to local files using kubectl.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+Get the SAT configuration file’s endpoint value.
+Note: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
ncn-m001# grep endpoint ~/.config/sat/sat.toml
+# endpoint = "https://rgw-vip.nmn"
+Get the sat-s3-credentials secret’s endpoint value.
ncn-m001# kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, change the SAT configuration file’s endpoint value to +match the secret’s.
+Copy SAT configurations to each manager node on the system.
+ncn-m001# for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+Note: Depending on how many manager nodes are on the system, the list of
+manager nodes may be different. This example assumes three manager nodes, where
+the configuration files must be copied from ncn-m001 to ncn-m002 and
+ncn-m003. Therefore, the list of hosts above is ncn-m002 and ncn-m003.
HPE service representatives use system revision information data to identify +systems in support cases.
+Set System Revision Information.
+Run sat setrev and follow the prompts to set the following site-specific values:
Tip: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. In other words, “System type” is EX-1C.
+ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Verify System Revision Information.
+Run sat showrev and verify the output shown in the “System Revision Information table.”
The following example shows sample table output.
+ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+The System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands
+used on the Cray XC platform. For more information on SAT commands, see SAT Command Overview.
Six Kibana Dashboards are included with SAT. They provide organized output for system health information.
+Four Grafana Dashboards are included with SAT. They display messages that are generated by the HSN (High Speed Network) and +are reported through Redfish.
+In CSM 1.3 and newer, the sat command is automatically available on all the
+Kubernetes NCNs. For more information, see SAT in CSM. Older
+versions of CSM do not have the sat command automatically available, and SAT
+must be installed as a separate product.
Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes manager nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the
+HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are
+similarities between SAT commands and xt commands used on the Cray XC platform.
The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using Podman, a daemonless container runtime. SAT runs on
+Kubernetes manager nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
+sat bash, followed by a sat command.sat command directly on a Kubernetes manager node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using interactive and non-interactive modes.
ncn-m001# sat bash
+(CONTAINER-ID)sat-container# sat status
+ncn-m001# sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, then use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes manager node, use sat-man on the manager node as shown in the following
+example.
ncn-m001# sat-man status
+A man page describing the SAT container environment is available on the Kubernetes manager nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
ncn-m001# man sat
+ncn-m001# man sat-podman
+Some SAT subcommands make requests to the Shasta services through the API
+gateway and thus require authentication to the API gateway in order to function.
+Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured. In order to use the SAT S3 bucket, the System Administrator must
+generate the S3 access key and secret keys and write them to a local file. This
+must be done on every Kubernetes manager node where SAT commands are run.
For more information on authentication requests, see System Security and +Authentication in the Cray System Management +Documentation. The following is a table +describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bmccreds |
+Requires authentication to the API gateway. | +sat-bmccreds |
+Set BMC passwords. | +
sat bootprep |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is done on ncn-m001 during the install. |
+sat-bootprep |
+Prepare to boot nodes with images and configurations. | +
sat bootsys |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwhist |
+Requires authentication to the API gateway. | +sat-hwhist |
+Report hardware component history. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat jobstat |
+Requires authentication to the API gateway. | +sat-jobstat |
+Check the status of jobs and applications. | +
sat k8s |
+Requires Kubernetes configuration and authentication, which is automatically configured on ncn-m001 during the install. |
+sat-k8s |
+Report on Kubernetes replica sets that have co-located (on the same node) replicas. | +
sat linkhealth |
+This command has been deprecated. | ++ | + |
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node XNames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat slscheck |
+Requires authentication to the API gateway. | +sat-slscheck |
+Perform a cross-check between SLS and HSM. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC XNames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, you must run the sat auth
+command. This command will prompt for a password on the command line. The
+username value is obtained from the following locations, in order of higher
+precedence to lower precedence:
--username global command-line option.username option in the api_gateway section of the config file at
+~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file
+will be obtained and saved to ~/.config/sat/tokens. Subsequent sat commands
+will determine the username the same way as sat auth described above and will
+use the token for that username if it has been obtained and saved by sat auth.
The host name in a command prompt indicates where the command must be run. The account that must run the command is +also indicated in the prompt.
+root or super-user account always has the # character at the end of the prompt and has the host name of the
+host in the prompt.root account is indicated with account@hostname>. A user account that is neither root nor crayadm is
+referred to as user.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run on one of the Kubernetes Manager servers. (Non-interactive) | +
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
Here are examples of the sat status command used by an administrator.
ncn-m001# sat status
+ncn-m001# sat bash
+(CONTAINER_ID) sat-container# sat status
+In CSM 1.3 and newer, the sat command is automatically available on all the Kubernetes NCNs, but it is still possible
+to install SAT as a separate product stream. Any version of SAT installed as a separate product stream overrides the
+sat command available in CSM. Installing the SAT product stream allows additional supporting components to be added:
An entry for SAT in the cray-product-catalog Kubernetes ConfigMap is only created by installing the SAT product
+stream. Otherwise, there will be no entry for this version of SAT in the output of sat showrev.
The sat-install-utility container image is only available with the full SAT product stream. This container image
+provides uninstall and downgrade functionality when used with the prodmgr command. (In SAT 2.3 and older, SAT was
+only available to install as a separate product stream. Because these versions were packaged with
+sat-install-utility, it is still possible to uninstall these versions of SAT.)
The docs-sat RPM package is only available with the full SAT product stream.
The sat-config-management git repository in Gitea (VCS) and thus the SAT layer of NCN CFS configuration is
+only available with the full SAT product stream.
If the SAT product stream is not installed, there will be no configuration content for SAT in VCS. Therefore, CFS
+configurations that apply to management NCNs (for example, management-23.5.0) should not include a SAT layer.
The SAT configuration layer modifies the permissions of files left over from prior installations of SAT, so that the
+Keycloak username that authenticates to the API gateway cannot be read by users other than root. Specifically, it
+it does the following:
Modifies the sat.toml configuration file which contains the username so that it is only readable by root.
Modifies the /root/.config/sat/tokens directory so that the directory is only readable by root. This is needed
+because the names of the files within the tokens directory contain the username.
Regardless of the SAT configuration being applied, passwords and the contents of the tokens are never readable by other +users. These permission changes only apply to files created by previous installations of SAT. In the current version of +SAT all files and directories are created with the appropriate permissions.
+Most sat subcommands depend on services or components from other products in the
+HPE Cray EX (Shasta) software stack. The following list shows these dependencies
+for each subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
+sat jobstatsat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nidSAT 2.2.16 was released on February 25th, 2022.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container image and Helm chartIt also added the following new components:
+sat-install-utility container imagecfs-config-util container imageThe following sections detail the changes in this release.
+sat Command Unavailable in sat bash ShellAfter launching a shell within the SAT container with sat bash, the sat
+command will not be found. For example:
(CONTAINER-ID) sat-container:~ # sat status
+bash: sat: command not found
+This can be resolved temporarily in one of two ways. /sat/venv/bin/ may be
+prepended to the $PATH environment variable:
(CONTAINER-ID) sat-container:~ # export PATH=/sat/venv/bin:$PATH
+(CONTAINER-ID) sat-container:~ # sat status
+Or, the file /sat/venv/bin/activate may be sourced:
(CONTAINER-ID) sat-container:~ # source /sat/venv/bin/activate
+(CONTAINER-ID) sat-container:~ # sat status
+sat bash ShellAfter launching a shell within the SAT container with sat bash, tab completion
+for sat commands does not work.
This can be resolved temporarily by sourcing the file
+/etc/bash_completion.d/sat-completion.bash:
source /etc/bash_completion.d/sat-completion.bash
+sat in Root Directorysat commands will not work if the current directory is /. For example:
ncn-m001:/ # sat --help
+Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: open /dev/console: operation not permitted: OCI runtime permission denied error
+To resolve, run sat in another directory.
sat in Config Directorysat commands will not work if the current directory is ~/.config/sat.
+For example:
ncn-m001:~/.config/sat # sat --help
+Error: /root/.config/sat: duplicate mount destination
+To resolve, run sat in another directory.
sat Commandssat bootprep automates the creation of CFS configurations, the build and
+customization of IMS images, and the creation of BOS session templates. For
+more information, see SAT Bootprep.sat slscheck performs a check for consistency between the System Layout
+Service (SLS) and the Hardware State Manager (HSM).sat bmccreds provides a simple interface for interacting with the System
+Configuration Service (SCSD) to set BMC Redfish credentials.sat hwhist displays hardware component history by XName (location) or by
+its Field-Replaceable Unit ID (FRUID). This command queries the Hardware
+State Manager (HSM) API to obtain this information. Since the sat hwhist
+command supports querying for the history of a component by its FRUID, the
+FRUID of components has been added to the output of sat hwinv.The following automation has been added to the install script, install.sh:
sat-config-import Kubernetes job, which is
+started when the sat-cfs-install Helm chart is deployed.ncn-personalization).The SAT product uploads additional information to the cray-product-catalog
+Kubernetes ConfigMap detailing the components it provides, including container
+(Docker) images, Helm charts, RPMs, and package repositories.
This information is used to support uninstall and downgrade of SAT product +versions moving forward.
+Beginning with the 2.2 release, SAT now provides partial support for the +uninstall and downgrade of the SAT product stream.
+For more information, see +Uninstall: Remove a Version of SAT and +Downgrade: Switch Between SAT Versions.
+sat statusA Subrole column has been added to the output of sat status. This allows you
+to easily differentiate between master, worker, and storage nodes in the
+management role, for example.
Hostname information from SLS has been added to sat status output.
Support for JSON-formatted output has been added to commands which currently
+support the --format option, such as hwinv, status, and showrev.
Many usability improvements have been made to multiple sat commands,
+mostly related to filtering command output. The following are some highlights:
--fields option to display only specific fields for subcommands which
+display tabular reports.--filter queries
+so that the first match is used, similar to --sort-by.--filter, --fields, and --reverse for summaries
+displayed by sat hwinv.sat hwinv.The default log level for stderr has been changed from “WARNING” to “INFO”. For
+more information, see Update SAT Logging.
With the command-line options --loglevel-stderr and --loglevel-file, the log
+level can now be configured separately for stderr and the log file.
The existing --loglevel option is now an alias for the --loglevel-stderr
+option.
The Podman wrapper script is the script installed at /usr/bin/sat on the
+master management NCNs by the cray-sat-podman RPM that runs the cray-sat
+container in podman. The following subsections detail improvements that were
+made to the wrapper script in this release.
cray-sat ContainerThe Podman wrapper script that launches the cray-sat container with podman
+has been modified to mount the user’s current directory and home directory into
+the cray-sat container to provide access to local files in the container.
The man page for the Podman wrapper script, which is accessed by typing man sat on a master management NCN, has been improved to document the following:
Fixed issues with redirecting stdout and stderr, and piping output to
+commands, such as awk, less, and more.
A new sat option has been added to configure the HTTP timeout length for
+requests to the API gateway. For more information, refer to sat-man sat.
sat bootsys ImprovementsMany improvements and fixes have been made to sat bootsys. The following are
+some highlights:
--excluded-ncns option, which can be used to omit NCNs
+from the platform-services and ncn-power stages in case they are
+inaccessible.sat bootsys shutdown now prompt the user to
+continue before proceeding. A new option, --disruptive, will bypass this.platform-services stage of sat bootsys boot.sat xname2nid Improvementssat xname2nid can now recursively expand slot, chassis, and cabinet XNames to
+a list of NIDs in those locations.
A new --format option has been added to sat xname2nid. It sets the output
+format to either “range” (the default) or “NID”. The “range” format displays NIDs
+in a compressed range format suitable for use with a workload manager like Slurm.
v2 HSM APIThe commands which interact with HSM (for example, sat status and sat hwinv)
+now use the v2 HSM API.
sat diag Limited to HSN Switchessat diag will now only operate against HSN switches by default. These are the
+only controllers that support running diagnostics with HMJTD.
sat showrev EnhancementsA column has been added to the output of sat showrev that indicates whether a
+product version is “active”. The definition of “active” varies across products,
+and not all products may set an “active” version.
For SAT, the active version is the one with its hosted-type package repository
+in Nexus set as the member of the group-type package repository in Nexus,
+meaning that it will be used when installing the cray-sat-podman RPM.
cray-sat Container Image Size ReductionThe size of the cray-sat container image has been approximately cut in half by
+leveraging multi-stage builds. This also improved the repeatability of the unit
+tests by running them in the container.
Minor bug fixes were made in cray-sat and in cray-sat-podman. For full
+change lists, refer to each repository’s CHANGELOG.md file.
The 2.3.4 version of the SAT product includes:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container imagesat-cfs-install Helm chartsat-install-utility container imagecfs-config-util container imagesat CommandsNone.
+When running sat commands, the current working directory is now mounted in the
+container as /sat/share, and the current working directory within the container
+is also /sat/share.
Files in the current working directory must be specified using relative paths to
+that directory, because the current working directory is always mounted on
+/sat/share. Absolute paths should be avoided, and paths that are outside of
+$HOME or $PWD are never accessible to the container environment.
The home directory is still mounted on the same path inside the container as it +is on the host.
+sat bootsysThe following options were added to sat bootsys.
--bos-limit--recursiveThe --bos-limit option passes a given limit string to a BOS session. The
+--recursive option specifies a slot or other higher-level component in the
+limit string.
sat bootprepThe --delete-ims-jobs option was added to sat bootprep run. It deletes IMS
+jobs after sat bootprep is run. Jobs are no longer deleted by default.
sat statussat status now includes information about nodes’ CFS configuration statuses,
+such as desired configuration, configuration status, and error count.
The output of sat status now splits different component types into different
+report tables.
The following options were added to sat status.
--hsm-fields, --sls-fields, --cfs-fields--bos-templateThe --hsm-fields, --sls-fields, --cfs-fields options limit the output
+columns according to specified CSM services.
The --bos-template option filters the status report according to the specified
+session template’s boot sets.
The following components were modified to be compatible with CSM 1.2.
+sat-cfs-install container image and Helm chartsat-install-utility container imageThe sat-ncn Ansible role provided by sat-cfs-install was modified to enable
+GPG checks on packages while leaving GPG checks disabled on repository metadata.
Updated urllib3 dependency to version 1.26.5 to mitigate CVE-2021-33503 and
+refreshed Python dependency versions.
Minor bug fixes were made in each of the repositories. For full change lists,
+refer to each repository’s CHANGELOG.md file.
The known issues listed under the SAT 2.2 release +were fixed.
+ + + + + +The 2.4.13 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.cfs-config-util container image.Because of installation refactoring efforts, the following two components +are no longer delivered with SAT:
+sat-cfs-install container imagesat-cfs-install Helm chartA version of the cray-sat container image is now included in CSM. For more
+information, see SAT in CSM.
The SAT install.sh script no longer uses a sat-cfs-install Helm chart and
+container image to upload its Ansible content to the sat-config-management
+repository in VCS. Instead, it uses Podman to run the cf-gitea-import container
+directly. Some of the benefits of this change include the following:
cray-sat container image and cray-sat-podman packagecray-sat Container Image and cray-sat-podman PackageIn older SAT releases, the sat wrapper script that was provided by the
+cray-sat-podman package installed on Kubernetes master NCNs included a
+hard-coded version of the cray-sat container image. As a result, every new
+version of the cray-sat image required a corresponding new version of the
+cray-sat-podman package.
In this release, this tight coupling of the cray-sat-podman package and the
+cray-sat container image was removed. The sat wrapper script provided
+by the cray-sat-podman package now looks for the version of the cray-sat
+container image in the /opt/cray/etc/sat/version file. This file is populated
+with the correct version of the cray-sat container image by the SAT layer of
+the CFS configuration that is applied to management NCNs. If the version file
+does not exist, the wrapper script defaults to the version of the cray-sat
+container image delivered with the latest version of CSM installed on the system.
The steps for performing NCN personalization as part of the SAT installation
+were moved out of the install.sh script and into a new
+update-mgmt-ncn-cfs-config.sh script that is provided in the SAT release
+distribution. The new script provides additional flexibility in how it modifies
+the NCN personalization CFS configuration for SAT. It can modify an existing CFS
+configuration by name, a CFS configuration being built in a JSON file, or an
+existing CFS configuration that applies to certain components.
sat bootprep FeaturesThe following new features were added to the sat bootprep command:
Variable substitutions using Jinja2 templates in certain fields of the
+sat bootprep input file
For more information, see +HPC CSM Software Recipe Variable Substitutions +and Dynamic Variable Substitutions.
+Schema version validation in the sat bootprep input files
For more information, see +Provide a Schema Version.
+Ability to look up images and recipes provided by products
+For more information, see +Define IMS Images.
+The schema of the sat bootprep input files was also changed to support these
+new features:
base key instead of under an ims key. The old ims
+key is deprecated.base.image_ref.
+You should no longer use the IMS name of the image on which it depends.image.ims.name, image.ims.id, or image.image_ref. Specifying a string
+value directly under the image key is deprecated.For more information on defining IMS images and BOS session templates in the
+sat bootprep input file, see Define IMS Images
+and Define BOS Session Templates.
sat swapThe sat swap command was updated to support swapping compute and UAN blades
+with sat swap blade. This functionality is described in the following processes
+of the Cray System Management Documentation:
v2A new v2 version of the Boot Orchestration Service (BOS) is available in CSM
+1.3.0. SAT has added support for BOS v2. This impacts the following commands
+that interact with BOS:
sat bootprepsat bootsyssat statusBy default, SAT uses BOS v1. However, you can choose the BOS version you want
+to use. For more information, see Change the BOS Version.
sat statusWhen using BOS v2, sat status outputs additional fields. These fields show
+the most recent BOS session, session template, booted image, and boot status for
+each node. An additional --bos-fields option was added to limit the output of
+sat status to these fields. The fields are not displayed when using BOS v1.
This is the first release of SAT built from open source code repositories. +As a result, build infrastructure was changed to use an external Jenkins instance, +and artifacts are now published to an external Artifactory instance. These +changes should not impact the functionality of the SAT product in any way.
+paramiko Python package version was updated from 2.9.2 to 2.10.1 to
+mitigate CVE-2022-24302.oauthlib Python package version was updated from 3.2.0 to 3.2.1 to
+mitigate CVE-2022-36087.SAT stores information used to authenticate to the API gateway with Keycloak.
+Token files are stored in the ~/.config/sat/tokens/ directory. Those files
+have always had permissions appropriately set to restrict them to be readable
+only by the user.
Keycloak usernames used to authenticate to the API gateway are stored in the
+SAT config file at /.config/sat/sat.toml. Keycloak usernames are also used in
+the file names of tokens stored in /.config/sat/tokens. As an additional
+security measure, SAT now restricts the permissions of the SAT config file
+to be readable and writable only by the user. It also restricts the tokens
+directory and the entire SAT config directory ~/.config/sat to be accessible
+only by the user. This prevents other users on the system from viewing
+Keycloak usernames used to authenticate to the API gateway.
sat init did not print a message confirming a new
+configuration file was created.sat showrev exited with a traceback if the file
+/opt/cray/etc/site_info.yaml existed but was empty. This could occur if the
+user exited sat setrev with Ctrl-C.sat bootsys man page, and added a
+description of the command stages.The 2.5.22 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.cfs-config-util container image.sat Commandssat jobstat allows you to access application and job data through the command
+line. It provides a table summarizing information for all jobs on the system.
sat bootprepA list-vars subcommand was added to sat bootprep.
It lists the variables available for use in bootprep input files at runtime.
+A --limit option was added to sat bootprep run.
It restricts the creation of CFS configurations, IMS images, and BOS session +templates into separate stages. For more information, see +Limit SAT Bootprep Run into Stages.
+sat bootprep now prompts individually for each CFS configuration that
+already exists.
sat bootprep can now filter images provided by a product by using a prefix.
This is useful when specifying the base of an image in a bootprep input +file. For more information, see +Define IMS Images.
+To support product names with hyphens, sat bootprep now converts hyphens to
+underscores within variables.
For more information, see +Hyphens in HPC CSM Software Recipe Variables.
+In sat bootprep input files, you can now render the value of the playbook
+property of CFS configuration layers with Jinja2 templates.
For more information, see +Values Supporting Jinja2 Template Rendering.
+Output was added to sat bootprep run that summarizes the CFS configurations,
+IMS images, and BOS session templates created.
For more information, see +Summary of SAT Bootprep Results.
+Improvements were made to the sat bootprep output when CFS configuration
+and BOS session templates are created.
sat bootsysreboot subcommand was added to sat bootsys. It uses BOS to reboot
+nodes in the bos-operations stage.--staged-session option was added to sat bootsys. It can be used to
+create staged BOS sessions. For more information, refer to Staging Changes
+with BOS in the Cray System Management Documentation.sat Commandsprodmgr, a version is no longer set as
+“active” in the product catalog. The “active” field was also removed from the
+output of sat showrev.sat status when using BOS
+version two.The new Install and Upgrade Framework (IUF) provides commands which install,
+upgrade, and deploy products with the help of sat bootprep on HPE Cray EX
+systems managed by Cray System Management (CSM). IUF capabilities are described
+in detail in the IUF section
+of the Cray System Management Documentation.
+The initial install and upgrade workflows described in the
+HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM
+(S-8052) detail when and how to use
+IUF with a new release of SAT or any other HPE Cray EX product.
Because IUF now handles NCN personalization, information about this process was +removed from the SAT documentation. Other sections in the documentation were +also revised to support the new Install and Upgrade Framework. For example, the +SAT Installation and SAT Upgrade sections of this +guide now provide details on software and configuration content specific to SAT. +The Cray System Management Documentation +will indicate when these sections should be referred to for detailed information.
+For more information on the relationship between sat bootprep and IUF, see
+SAT and IUF.
By default, SAT now uses version two of the Boot Orchestration Service (BOS).
+This change to BOS v2 impacts the following commands that interact with BOS:
sat bootprepsat bootsyssat statusIf needed, you can choose the BOS version you want to use. For more information, +see Change the BOS Version.
+sat python package and CLI from
+2021.10.8 to 2022.12.7 to resolve CVE-2022-23491.sat-install-utility container image
+from 2021.5.30 to 2022.12.7 to resolve CVE-2022-23491.sat init from creating a configuration file in
+the current directory when not prefixed with ./.sat status failed with a traceback when using BOS
+version two and reported components whose most recent image did not exist.sat container could contain a different
+version of kubectl than the version found in CSM.sat bootprep and
+sat swap blade.Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchFor more information on each of these commands, see the +SAT Command Overview and the table +of commands in the Authenticate SAT Commands +section of this document.
+ + + + + +We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes management cluster (workers and
+masters). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Config File Location ChangeThe default location of the SAT config file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own config files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment
+variable.
Additionally, if a config file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies
+option.--list-node-accels option.
+The count of node accelerators is also included for each node.--list-node-accel-risers option. The count of node accelerator risers is also
+included for each node.--list-node-hsn-nics option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For more +information, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding config-file
+option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding config file options were
+deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
We released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named ncn-personalization.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-Diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+programs on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and config file options +have been removed. For more information, see +Remove Obsolete Configuration File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then
+displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes masters and workers.
sat firmware.cray-sat container image.sat firmware command.This procedure can be used to uninstall a version of SAT.
+prodmgr.prodmgr command is available.Use sat showrev to list versions of SAT.
ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+-------------------+-----------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+-------------------+-----------------------+
+| sat | 2.3.3 | - | - |
+| sat | 2.2.10 | - | - |
++--------------+-----------------+-------------------+-----------------------+
+Use prodmgr to uninstall a version of SAT.
This command will do three things:
+cray-product-catalog Kubernetes ConfigMap, so that it will no longer show up
+in the output of sat showrev.ncn-m001# prodmgr uninstall sat 2.2.10
+Repository sat-2.2.10-sle-15sp2 has been removed.
+Removed Docker image cray/cray-sat:3.9.0
+Removed Docker image cray/sat-cfs-install:1.0.2
+Removed Docker image cray/sat-install-utility:1.4.0
+Deleted sat-2.2.10 from product catalog.
+This procedure can be used to downgrade the active version of SAT.
+prodmgr command is available.Use sat showrev to list versions of SAT.
ncn-m001# sat showrev --products --filter product_name=sat
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------------------+-----------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+--------------------+-----------------------+
+| sat | 2.3.3 | - | - |
+| sat | 2.2.10 | - | - |
++--------------+-----------------+--------------------+-----------------------+
+Use prodmgr to switch to a different version of SAT.
This command will do two things:
+2.2.10
+sets the repository sat-2.2.10-sle-15sp2 as the only member of the sat-sle-15sp2 group.management-23.5.0). Specifically, it will ensure that the layer refers to the version of SAT CFS
+configuration content associated with the version of SAT to which you are switching.ncn-m001# prodmgr activate sat 2.5.15
+Repository sat-2.5.15-sle-15sp4 is now the default in sat-sle-15sp4.
+Updated CFS configurations: [management-23.5.0]
+Apply the modified CFS configuration to the management NCNs.
+At this point, Nexus package repositories have been modified to set a +particular package repository as active, but the SAT package may not have +been updated on management NCNs.
+To ensure that management NCNs have been updated to use the active SAT +version, follow the Procedure to Apply CFS Configuration.
+Set an environment variable that refers to the name of the CFS configuration +to be applied to the management NCNs.
+ncn-m001# export CFS_CONFIG_NAME="management-23.5.0"
+Note: Refer to the output from the prodmgr activate command to find
+the name of the modified CFS configuration. If more than one CFS configuration
+was modified, use the first one.
INFO: Successfully saved CFS configuration "management-23.5.0"
+Obtain the name of the CFS configuration layer for SAT and save it in an +environment variable:
+ncn-m001# export SAT_LAYER_NAME=$(cray cfs configurations describe $CFS_CONFIG_NAME --format json \
+ | jq -r '.layers | map(select(.cloneUrl | contains("sat-config-management.git")))[0].name')
+Create a CFS session that executes only the SAT layer of the given CFS +configuration.
+The --configuration-limit option limits the configuration session to run
+only the SAT layer of the configuration.
ncn-m001# cray cfs sessions create --name "sat-session-${CFS_CONFIG_NAME}" --configuration-name \
+ "${CFS_CONFIG_NAME}" --configuration-limit "${SAT_LAYER_NAME}"
+Monitor the progress of the CFS session.
+Set an environment variable to name of the Ansible container within the pod +for the CFS session:
+ncn-m001# export ANSIBLE_CONTAINER=$(kubectl get pod -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME} -o json \
+ -o json | jq -r '.items[0].spec.containers | map(select(.name | contains("ansible"))) | .[0].name')
+Next, get the logs for the Ansible container.
+ncn-m001# kubectl logs -c $ANSIBLE_CONTAINER --tail 100 -f -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME}
+Ansible plays, which are run by the CFS session, will install SAT on all the +master management NCNs on the system. A summary of results can be found at +the end of the log output. The following example shows a successful session.
+...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Note: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+Verify that SAT was successfully configured.
+If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
Note: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
ncn-m001# sat --version
+sat 3.7.0
+Note: Upon first running sat, you may see additional output while the sat
+container image is downloaded. This will occur the first time sat is run on
+each manager NCN. For example, if you run sat for the first time on ncn-m001
+and then for the first time on ncn-m002, you will see this additional output
+both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+Stop the typescript.
+ncn-m001# exit
+SAT version x.y.z is now installed and configured:
The previous procedure is not always necessary because the CFS Batcher service +automatically detects configuration changes and will automatically create new +sessions to apply configuration changes according to certain rules. For more +information on these rules, refer to Configuration Management with +the CFS Batcher in the Cray System Management Documentation.
+The main scenario in which the CFS batcher will not automatically re-apply the +SAT layer is when the commit hash of the sat-config-management git repository +has not changed between SAT versions. The previous procedure ensures the +configuration is re-applied in all cases, and it is harmless if the batcher has +already applied an updated configuration.
+ + + + + +The Install and Upgrade Framework (IUF) provides commands which install, +upgrade, and deploy products on systems managed by CSM. IUF capabilities are +described in detail in the IUF +section of the +Cray System Management Documentation. +The initial install and upgrade workflows described in the +HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM +(S-8052) detail when and how to use +IUF with a new release of SAT or any other HPE Cray EX product.
+This document does not replicate install, upgrade, or deployment procedures +detailed in the Cray System Management +Documentation. This document provides +details regarding software and configuration content specific to SAT which is +needed when installing, upgrading, or deploying a SAT release. The Cray +System Management Documentation will +indicate when sections of this document should be referred to for detailed +information.
+IUF will perform the following tasks for a release of SAT.
+deliver-product stage:
+update-vcs-config stage:
+update-cfs-config stage:
+prepare-images stage:
+management-nodes-rollout stage:
+IUF uses a variety of CSM and SAT tools when performing these tasks. The IUF +section of the +Cray System Management Documentation +describes how to use these tools directly if it is desirable to use them +instead of IUF.
+This section describes SAT details that an administrator must be aware of +before running IUF stages. Entries are prefixed with Information if no +administrative action is required or Action if an administrator needs +to perform tasks outside of IUF.
+Information: This stage is only run if a VCS working branch is specified for +SAT. By default, SAT does not create or specify a VCS working branch.
+Information: This stage only applies to the management configuration and +not to the managed configuration.
+Information: This stage only applies to management images and not to +managed images.
+After upgrading SAT with IUF, it is recommended that you complete the following +procedures before using SAT:
+ +...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being upgraded.After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
+In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accommodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
ncn-m001:~ # grep -A 3 logging ~/.config/sat/sat.toml
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful to set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checkssat sensorsThe following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xnamesat xname2nidsat swapHPE service representatives use system revision information data to identify +systems in support cases.
+This procedure is not required if SAT was upgraded from 2.1 (Shasta v1.5) +or later. It is required if SAT was upgraded from 2.0 (Shasta v1.4) or +earlier.
+Set System Revision Information.
+Run sat setrev and follow the prompts to set the following site-specific values:
Tip: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. In other words, “System type” is EX-1C.
+ncn-m001# sat setrev
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Verify System Revision Information.
+Run sat showrev and verify the output shown in the “System Revision Information table.”
The following example shows sample table output.
+ncn-m001# sat showrev
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+By default, SAT uses Boot Orchestration Service (BOS) version two (v2). You can
+select the BOS version to use for individual commands with the --bos-version
+option. For more information on this option, refer to the man page for a specific
+command.
You can also configure the BOS version to use in the SAT config file. Do this
+under the api_version setting in the bos section of the config file. If
+the system is using an existing SAT config file from an older version of SAT,
+the bos section might not exist. In that case, add the bos section with the
+BOS version desired in the api_version setting.
Find the SAT config file at ~/.config/sat/sat.toml, and look for a section
+like this:
[bos]
+api_version = "v2"
+In this example, SAT is using BOS version "v2".
Change the line specifying the api_version to the BOS version desired (for
+example, "v1").
[bos]
+api_version = "v1"
+If applicable, uncomment the api_version line.
If the system is using an existing SAT config file from a recent version of
+SAT, the api_version line might be commented out like this:
[bos]
+# api_version = "v2"
+If the line is commented out, SAT will still use the default BOS
+version. To ensure a different BOS version is used, uncomment the
+api_version line by removing # at the beginning of the line.
The Install and Upgrade Framework (IUF) provides commands which install,
+upgrade, and deploy products on systems managed by CSM with the help of
+sat bootprep. Outside of IUF, it is uncommon to use sat bootprep.
+For more information on IUF, see the
+IUF section of
+the Cray System Management Documentation.
+For more information on sat bootprep, see SAT Bootprep.
Both IUF and sat bootprep allow variable substitutions into the default HPC
+CSM Software Recipe bootprep input files. The default variables of the HPC
+CSM Software Recipe are available in a product_vars.yaml file. To override
+the default variables, specify any site variables in a site_vars.yaml file.
+Variables are sourced from the command line, any variable files directly
+provided, and the HPC CSM Software Recipe files used, in that order.
IUF also has special session variables internal to the iuf command that
+override any matching entries. Session variables are the set of product and
+version combinations being installed by the current IUF activity, and they are
+found inside IUF’s internal session_vars.yaml file. For more information on
+IUF and variable substitutions, see the
+IUF section of
+the Cray System Management Documentation.
When using sat bootprep outside of IUF, you might encounter problems
+substituting variables into the default bootprep input files. Complex variables
+like "{{ working_branch }}" cannot be completely resolved outside of IUF and
+its internal session variables. Thus, the default product_vars.yaml file is
+unusable with only the sat bootprep command when variables like
+"{{ working_branch }}" are used. To work around this limitation if you are
+substituting complex variables, use the internal IUF session_vars.yaml file
+with sat bootprep and the default bootprep input files.
Find the session_vars.yaml file from the most recent IUF activity on the
+system.
This process is documented in the upgrade prerequisites procedure of the +Cray System Management Documentation. For more information, see steps 1-6 of +Stage 0.3 - Option 2.
+Use the session_vars.yaml file to substitute variables into the default
+bootprep input files.
ncn-m001# sat bootprep run --vars-file session_vars.yaml
+The sat bootprep run command uses information from the bootprep input files
+to create CFS configurations, IMS images, and BOS session templates. To restrict
+this creation into separate stages, use the --limit option and list whether
+you want to create configurations, images, session_templates, or some
+combination of these. IUF uses the --limit option in this way to install,
+upgrade, and deploy products on a system in stages. For example, to create only
+CFS configurations, run the following command used by the IUF update-cfs-config
+stage:
ncn-m001# sat bootprep run --limit configurations example-bootprep-input-file.yaml
+INFO: Validating given input file example-bootprep-input-file.yaml
+INFO: Input file successfully validated against schema
+INFO: Creating 3 CFS configurations
+...
+INFO: Skipping creation of IMS images based on value of --limit option.
+INFO: Skipping creation of BOS session templates based on value of --limit option.
+To create only IMS images and BOS session templates, run the following command
+used by the IUF prepare-images stage:
ncn-m001# sat bootprep run --limit images --limit session_templates example-bootprep-input-file.yaml
+INFO: Validating given input file example-bootprep-input-file.yaml
+INFO: Input file successfully validated against schema
+INFO: Skipping creation of CFS configurations based on value of --limit option.
+SAT provides an automated solution for creating CFS configurations, building
+and configuring images in IMS, and creating BOS session templates. The
+solution is based on a given input file that defines how those configurations,
+images, and session templates should be created. This automated process centers
+around the sat bootprep command. Man page documentation for sat bootprep
+can be viewed similar to other SAT commands.
ncn-m001# sat-man sat-bootprep
+The sat bootprep command helps the Install and Upgrade Framework (IUF)
+install, upgrade, and deploy products on systems managed by CSM. Outside of IUF,
+it is uncommon to use sat bootprep. For more information on this relationship,
+see SAT and IUF. For more information on IUF, see the
+IUF section of
+the Cray System Management Documentation.
sat bootprep is used to create CFS configurations, build and
+rename IMS images, and create BOS session templates which tie the
+configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
+including (but not limited to) performing BOS operations (such as creating BOS
+sessions), powering on and off cabinets, and checking the state of the system
+prior to shutdown.
The input file provided to sat bootprep is a YAML-formatted file containing
+information which CFS, IMS, and BOS use to create configurations, images, and
+BOS session templates respectively. Writing and modifying these input files is
+the main task associated with using sat bootprep. An input file is composed of
+three main sections, one each for configurations, images, and session templates.
+These sections may be specified in any order, and any of the sections may be
+omitted if desired.
The sat bootprep input file is validated against a versioned schema
+definition. The input file should specify the version of the schema with which
+it is compatible under a schema_version key. For example:
---
+schema_version: 1.0.2
+The current sat bootprep input file schema version can be viewed with the
+following command:
ncn-m001# sat bootprep view-schema | grep '^version:'
+version: '1.0.2'
+The sat bootprep run command validates the schema version specified
+in the input file. The command also makes sure that the schema version
+of the input file is compatible with the schema version understood by the
+current version of sat bootprep. For more information on schema version
+validation, refer to the schema_version property description in the bootprep
+input file schema. For more information on viewing the bootprep input file
+schema in either raw form or user-friendly HTML form, see View SAT Bootprep
+Schema.
The default HPC CSM Software Recipe bootprep input files provided by the
+hpc-csm-software-recipe release distribution already contain the correct
+schema version.
The CFS configurations are defined under a configurations key. Under this
+key, you can list one or more configurations to create. For each
+configuration, give a name in addition to the list of layers that
+comprise the configuration.
Each layer can be defined by a product name and optionally a version number,
+commit hash, or branch in the product’s configuration repository. If this
+method is used, the layer is created in CFS by looking up relevant configuration
+information (including the configuration repository and commit information) from
+the cray-product-catalog Kubernetes ConfigMap as necessary. A version may be
+supplied. However, if it is absent, the version is assumed to be the latest
+version found in the cray-product-catalog.
Alternatively, a configuration layer can be defined by explicitly referencing
+the desired configuration repository. You must then specify the intended version
+of the Ansible playbooks by providing a branch name or commit hash with branch
+or commit.
The following example shows a CFS configuration with two layers. The first +layer is defined in terms of a product name and version, and the second layer +is defined in terms of a Git clone URL and branch:
+---
+configurations:
+- name: example-configuration
+ layers:
+ - name: example-product
+ playbook: example.yml
+ product:
+ name: example
+ version: 1.2.3
+ - name: another-example-product
+ playbook: another-example.yml
+ git:
+ url: "https://vcs.local/vcs/another-example-config-management.git"
+ branch: main
+When sat bootprep is run against an input file, a CFS configuration is created
+corresponding to each configuration in the configurations section. For
+example, the configuration created from an input file with the layers listed
+above might look something like the following:
{
+ "lastUpdated": "2022-02-07T21:47:49Z",
+ "layers": [
+ {
+ "cloneUrl": "https://vcs.local/vcs/example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "example product",
+ "playbook": "example.yml"
+ },
+ {
+ "cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "another example product",
+ "playbook": "another-example.yml"
+ }
+ ],
+ "name": "example-configuration"
+}
+The IMS images are defined under an images key. Under the images key, the
+user may define one or more images to be created in a list. Each element of the
+list defines a separate IMS image to be built and/or configured. Images must
+contain a name key and a base key.
The name key defines the name of the resulting IMS image. The base key
+defines the base image to be configured or the base recipe to be built and
+optionally configured. One of the following keys must be present under the
+base key:
ims key to specify an existing image or recipe in IMS.product key to specify an image or recipe provided by a particular
+version of a product. If a product provides more than one image or recipe,
+a filter string prefix must be specified to select one.image_ref key to specify another image from the input file
+using its ref_name.Images may also contain the following keys:
+configuration key to specify a CFS configuration with which to
+customize the built image. If a configuration is specified, then configuration
+groups must also be specified using the configuration_group_names key.ref_name key to specify a unique name that can refer to this image
+within the input file in other images or in session templates. The ref_name
+key allows references to images from the input file that have dynamically
+generated names as described in
+Dynamic Variable Substitutions.description key to describe the image in the bootprep input file.
+Note that this key is not currently used.Here is an example of an image using an existing IMS recipe as its base. This
+example builds an IMS image from that recipe. It then configures it with
+a CFS configuration named example-compute-config. The example-compute-config
+CFS configuration can be defined under the configurations key in the same
+input file, or it can be an existing CFS configuration. Running sat bootprep
+against this input file results in an image named example-compute-image.
images:
+- name: example-compute-image
+ description: >
+ An example compute node image built from an existing IMS recipe.
+ base:
+ ims:
+ name: example-compute-image-recipe
+ type: recipe
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+Here is an example showing the definition of two images. The first image is
+built from a recipe provided by the cos product. The second image uses the
+first image as a base and configures it with a configuration named
+example-compute-config. The value of the first image’s ref_name key is used
+in the second image’s base.image_ref key to specify it as a dependency.
+Running sat bootprep against this input file results in two images, the
+first named example-cos-image and the second named example-compute-image.
images:
+- name: example-cos-image
+ ref_name: example-cos-image
+ description: >
+ An example image built from a recipe provided by the COS product.
+ base:
+ product:
+ name: cos
+ version: 2.3.101
+ type: recipe
+- name: example-compute-image
+ description: >
+ An example image built from a recipe provided by the COS product.
+ base:
+ image_ref: example-cos-image
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+Here is an example of three IMS images built from the Kubernetes image and the
+Ceph storage image provided by the csm product. This example uses a filter
+string prefix to select from the multiple images provided by the CSM product.
+The first two IMS images in the example find any image from the specified csm
+product version whose name starts with secure-kubernetes. The third image in
+the example finds any csm image whose name starts with secure-storage-ceph.
+All three images are then configured with a configuration named
+example-management-config. Running sat bootprep against this input file
+results in three IMS images named worker-example-csm-image,
+master-example-csm-image, and storage-example-csm-image.
images:
+- name: worker-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-kubernetes
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Worker
+
+- name: master-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-kubernetes
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Master
+
+- name: storage-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-storage-ceph
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Storage
+The BOS session templates are defined under the session_templates key. Each
+session template must provide values for the name, image, configuration,
+and bos_parameters keys. The name key defines the name of the resulting BOS
+session template. The image key defines the image to use in the BOS session
+template. One of the following keys must be present under the image key:
ims key to specify an existing image or recipe in IMS.image_ref key to specify another image from the input file
+using its ref_name.The configuration key defines the CFS configuration specified
+in the BOS session template.
The bos_parameters key defines parameters that are passed through directly to
+the BOS session template. The bos_parameters key should contain a boot_sets
+key, and each boot set in the session template should be specified under
+boot_sets. Each boot set can contain the following keys, all of
+which are optional:
kernel_parameters key to specify the parameters passed to the kernel on the command line.network key to specify the network over which the nodes boot.node_list key to specify the nodes to add to the boot set.node_roles_groups key to specify the HSM roles to add to the boot set.node_groups key to specify the HSM groups to add to the boot set.rootfs_provider key to specify the root file system provider.rootfs_provider_passthrough key to specify the parameters to add to the rootfs=
+kernel parameter.As mentioned above, the parameters under bos_parameters are passed through
+directly to BOS. For more information on the properties of a BOS boot set,
+refer to BOS Session Templates in the Cray
+System Management Documentation.
Here is an example of a BOS session template that refers to an existing IMS +image by name:
+session_templates:
+- name: example-session-template
+ image:
+ ims:
+ name: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+Here is an example of a BOS session template that refers to an image from the
+input file by its ref_name. This requires that an image defined in the input
+file specifies example-image as the value of its ref_name key.
session_templates:
+- name: example-session-template
+ image:
+ image_ref: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+The sat bootprep command takes any variables you provide and substitutes them
+into the input file. Variables are sourced from the command line, any variable
+files directly provided, and the HPC CSM Software Recipe files used, in that
+order. When you provide values through a variable file, sat bootprep
+substitutes the values with Jinja2 template syntax. The HPC CSM Software Recipe
+provides default variables in a product_vars.yaml variable file. This file
+defines information about each HPC software product included in the recipe.
You will primarily substitute variables into the default HPC CSM Software Recipe
+bootprep input files through IUF. However, variable files can also be given to
+sat bootprep directly from IUF’s use of the recipe. If you do use variables
+directly with sat bootprep, you might encounter some limitations. For more
+information on SAT variable limitations, see SAT and IUF.
+For more information on IUF and variable substitutions, see the
+IUF section of
+the Cray System Management Documentation.
You can view a listing of the default HPC CSM Software Recipe variables and
+their values by running sat bootprep list-vars. For more information on
+options that can be used with the list-vars subcommand, refer to the man page
+for the sat bootprep subcommand.
By default, the sat bootprep command uses the variables from the latest
+installed version of the HPC CSM Software Recipe. However, you can override
+this with the --recipe-version command line argument to sat bootprep run.
For example, to explicitly select the 22.11.0 version of the HPC CSM Software
+Recipe default variables, specify --recipe-version 22.11.0:
ncn-m001# sat bootprep run --recipe-version 22.11.0 compute-and-uan-bootprep.yaml
+The entire sat bootprep input file is not rendered by the Jinja2 template
+engine. Jinja2 template rendering of the input file is performed individually
+for each supported value. The values of the following keys in the bootprep
+input file support rendering as a Jinja2 template and thus support variables:
name key of each configuration under the configurations key.layers key in a
+configuration:
+nameplaybookgit.branchproduct.versionproduct.branchimages key:
+namebase.product.versionconfigurationsession_templates key:
+nameconfigurationYou can use Jinja2 built-in filters in values of any of the keys listed above. +In addition, Python string methods can be called on the string variables.
+Variable names with hyphens are not allowed in Jinja2 expressions because they
+are parsed as an arithmetic expression instead of a single variable. To support
+product names with hyphens, sat bootprep converts hyphens to underscores in
+all top-level keys of the default HPC CSM Software Recipe variables. It also
+converts any variables sourced from the command line or any variable files
+you provide directly. When referring to a variable with hyphens in the bootprep
+input file, keep this in mind. For example, to refer to the product version
+variable for slingshot-host-software in the bootprep input file, write
+"{{slingshot_host_software.version}}".
The following example bootprep input file shows how a variable of a COS version +can be used in an input file that creates a CFS configuration for computes. +Only one layer is shown for brevity.
+---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: cos-compute-{{cos.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: "{{cos.working_branch}}"
+Note: When the value of a key in the bootprep input file is a Jinja2 +expression, it must be quoted to pass YAML syntax checking.
+Jinja2 expressions can also use filters and Python’s built-in string methods to
+manipulate the variable values. For example, suppose only the major and minor
+components of a COS version are to be used in the branch name for the COS
+layer of the CFS configuration. You can use the split string method to
+achieve this as follows:
---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: cos-compute-{{cos.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: integration-{{cos.version.split('.')[0]}}-{{cos.version.split('.')[1]}}
+Additional variables are available besides the default variables provided by +the HPC CSM Software Recipe. (For more information, see HPC CSM Software +Recipe Variable Substitutions.) +These additional variables are dynamic because their values are determined +at run-time based on the context in which they appear. Available dynamic +variables include the following:
+The variable base.name can be used in the name of an image under the
+images key. The value of this variable is the name of the IMS image or
+recipe used as the base of this image.
The variable image.name can be used in the name of a session template
+under the session_templates key. The value of this variable is the name of
+the IMS image used in this session template.
Note: The name of a session template is restricted to 45 characters. Keep
+this in mind when using image.name in the name of a session template.
These variables reduce the need to duplicate values throughout the sat bootprep input file and make the following use cases possible:
This section provides an example bootprep input file. It also gives +instructions for obtaining the default bootprep input files delivered +with a release of the HPC CSM Software Recipe.
+The following bootprep input file provides an example of using most of the +features described in previous sections. It is not intended to be a complete +bootprep file for the entire CSM product.
+---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: cos-compute-{{cos.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: cos
+ version: "{{cos.version}}"
+ branch: "{{cos.working_branch}}"
+ - name: cpe-pe_deploy-{{cpe.working_branch}}
+ playbook: pe_deploy.yml
+ product:
+ name: cpe
+ version: "{{cpe.version}}"
+ branch: "{{cpe.working_branch}}"
+
+images:
+- name: "{{default.note}}{{base.name}}{{default.suffix}}"
+ ref_name: base_cos_image
+ base:
+ product:
+ name: cos
+ type: recipe
+ version: "{{cos.version}}"
+
+- name: "compute-{{base.name}}"
+ ref_name: compute_image
+ base:
+ image_ref: base_cos_image
+ configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ configuration_group_names:
+ - Compute
+
+session_templates:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ image:
+ image_ref: compute_image
+ configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ bos_parameters:
+ boot_sets:
+ compute:
+ kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
+ node_roles_groups:
+ - Compute
+ rootfs_provider_passthrough: "dvs:api-gw-service-nmn.local:300:hsn0,nmn0:0"
+Default bootprep input files are delivered by the HPC CSM Software Recipe
+product. You can access these files by cloning the hpc-csm-software-recipe
+repository, as described in the Accessing sat bootprep files process of
+the Cray System Management
+Documentation. Find the
+default input files in the bootprep directory of the cloned repository:
ncn-m001# ls bootprep/
+The sat bootprep generate-example command was not updated for
+recent bootprep schema changes. It is recommended that you instead use the
+default bootprep input files described in Access Default Bootprep Input
+Files. The sat bootprep generate-example command will be updated in a future release of SAT.
The sat bootprep run command uses information from the bootprep input file to
+create CFS configurations, IMS images, and BOS session templates. For easy
+reference, the command also includes output summarizing the final creation
+results. The following example shows a sample table output.
ncn-m001# sat bootprep run
+...
+################################################################################
+CFS configurations
+################################################################################
++------------------+
+| name |
++------------------+
+| example-config-1 |
+| example-config-2 |
++------------------+
+################################################################################
+IMS images
+################################################################################
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+| name | preconfigured_image_id | final_image_id | configuration | configuration_group_names |
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+| example-image | c1bcaf00-109d-470f-b665-e7b37dedb62f | a22fb912-22be-449b-a51b-081af2d7aff6 | example-config | Compute |
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+################################################################################
+BOS session templates
+################################################################################
++------------------+----------------+
+| name | configuration |
++------------------+----------------+
+| example-template | example-config |
++------------------+----------------+
+The contents of the YAML input files used by sat bootprep must conform to a
+schema which defines the structure of the data. The schema definition is written
+using the JSON Schema format. (Although the format is named “JSON Schema”, the
+schema itself is written in YAML as well.) More information, including introductory
+materials and a formal specification of the JSON Schema metaschema, can be found
+on the JSON Schema website.
To view the exact schema specification, run sat bootprep view-schema.
ncn-m001# sat bootprep view-schema
+---
+$schema: "https://json-schema.org/draft/2020-12/schema"
+...
+title: Bootprep Input File
+description: >
+ A description of the set of CFS configurations to create, the set of IMS
+ images to create and optionally customize with the defined CFS configurations,
+ and the set of BOS session templates to create that reference the defined
+ images and configurations.
+type: object
+additionalProperties: false
+properties:
+ ...
+The raw schema definition can be difficult to understand without experience
+working with JSON Schema specifications. For this reason, a feature is included
+with sat bootprep that generates user-friendly HTML documentation for the input
+file schema. This HTML documentation can be browsed with your preferred web
+browser.
Create a documentation tarball using sat bootprep.
ncn-m001# sat bootprep generate-docs
+INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
+An alternate output directory can be specified with the --output-dir
+option. The generated tarball is always named bootprep-schema-docs.tar.gz.
ncn-m001# sat bootprep generate-docs --output-dir /tmp
+INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
+From another machine, copy the tarball to a local directory.
+another-machine$ scp root@ncn-m001:bootprep-schema-docs.tar.gz .
+Extract the contents of the tarball and open the contained index.html.
another-machine$ tar xzvf bootprep-schema-docs.tar.gz
+x bootprep-schema-docs/
+x bootprep-schema-docs/index.html
+x bootprep-schema-docs/schema_doc.css
+x bootprep-schema-docs/schema_doc.min.js
+another-machine$ open bootprep-schema-docs/index.html
+Some SAT subcommands make requests to the HPE Cray EX services through the API
+gateway and thus require authentication to the API gateway in order to function.
+Other SAT subcommands use the Kubernetes API. Some sat commands require S3 to
+be configured. In order to use the SAT S3 bucket, the System Administrator must
+generate the S3 access key and secret keys and write them to a local file. This
+must be done on every Kubernetes control plane node where SAT commands are run.
For more information on authentication requests, see System Security and +Authentication in the Cray System Management +Documentation. The following is a table +describing SAT commands and the types of authentication they require.
+| SAT Subcommand | +Authentication/Credentials Required | +Man Page | +Description | +
|---|---|---|---|
sat auth |
+Responsible for authenticating to the API gateway and storing a token. | +sat-auth |
+Authenticate to the API gateway and save the token. | +
sat bmccreds |
+Requires authentication to the API gateway. | +sat-bmccreds |
+Set BMC passwords. | +
sat bootprep |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is done on ncn-m001 during the install. |
+sat-bootprep |
+Prepare to boot nodes with images and configurations. | +
sat bootsys |
+Requires authentication to the API gateway. Requires Kubernetes configuration and authentication, which is configured on ncn-m001 during the install. Some stages require passwordless SSH to be configured to all other NCNs. Requires S3 to be configured for some stages. |
+sat-bootsys |
+Boot or shutdown the system, including compute nodes, application nodes, and non-compute nodes (NCNs) running the management software. | +
sat diag |
+Requires authentication to the API gateway. | +sat-diag |
+Launch diagnostics on the HSN switches and generate a report. | +
sat firmware |
+Requires authentication to the API gateway. | +sat-firmware |
+Report firmware version. | +
sat hwhist |
+Requires authentication to the API gateway. | +sat-hwhist |
+Report hardware component history. | +
sat hwinv |
+Requires authentication to the API gateway. | +sat-hwinv |
+Give a listing of the hardware of the HPE Cray EX system. | +
sat hwmatch |
+Requires authentication to the API gateway. | +sat-hwmatch |
+Report hardware mismatches. | +
sat init |
+None | +sat-init |
+Create a default SAT configuration file. | +
sat jobstat |
+Requires authentication to the API gateway. | +sat-jobstat |
+Check the status of jobs and applications. | +
sat k8s |
+Requires Kubernetes configuration and authentication, which is automatically configured on ncn-m001 during the install. |
+sat-k8s |
+Report on Kubernetes replica sets that have co-located (on the same node) replicas. | +
sat linkhealth |
+This command has been deprecated. | ++ | + |
sat nid2xname |
+Requires authentication to the API gateway. | +sat-nid2xname |
+Translate node IDs to node XNames. | +
sat sensors |
+Requires authentication to the API gateway. | +sat-sensors |
+Report current sensor data. | +
sat setrev |
+Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-setrev |
+Set HPE Cray EX system revision information. | +
sat showrev |
+Requires API gateway authentication in order to query the Interconnect from HSM. Requires S3 to be configured for site information such as system name, serial number, install date, and site name. | +sat-showrev |
+Print revision information for the HPE Cray EX system. | +
sat slscheck |
+Requires authentication to the API gateway. | +sat-slscheck |
+Perform a cross-check between SLS and HSM. | +
sat status |
+Requires authentication to the API gateway. | +sat-status |
+Report node status across the HPE Cray EX system. | +
sat swap |
+Requires authentication to the API gateway. | +sat-swap |
+Prepare HSN switch or cable for replacement and bring HSN switch or cable into service. | +
sat xname2nid |
+Requires authentication to the API gateway. | +sat-xname2nid |
+Translate node and node BMC XNames to node IDs. | +
sat switch |
+This command has been deprecated. It has been replaced by sat swap. |
++ | + |
In order to authenticate to the API gateway, run the sat auth
+command. This command will prompt for a password on the command line. The
+username value is obtained from the following locations, in order of higher
+precedence to lower precedence:
--username global command-line option.username option in the api_gateway section of the configuration file
+at ~/.config/sat/sat.toml.sat command.If credentials are entered correctly when prompted by sat auth, a token file
+will be obtained and saved to ~/.config/sat/tokens. Subsequent sat commands
+will determine the username the same way as sat auth described above and will
+use the token for that username if it has been obtained and saved by sat auth.
Most sat subcommands depend on services or components from other products in the
+HPE Cray EX software stack. The following list shows these dependencies for each
+subcommand. Each service or component is listed under the product it belongs to.
sat authsat bmccredssat bootprepsat bootsyssat diagsat firmwaresat hwhistsat hwinvsat hwmatchsat initNone
+sat jobstatsat k8ssat nid2xnamesat sensorssat setrevsat showrevsat slschecksat statussat swapsat switchDeprecated: See sat swap
sat xname2nidThe System Admin Toolkit (SAT) is designed to assist administrators with common tasks, such as troubleshooting and +querying information about the HPE Cray EX System and its components, system boot and shutdown, and replacing hardware +components.
+SAT offers a command line utility which uses subcommands. There are similarities between SAT commands and xt commands
+used on the Cray XC platform. For more information on SAT commands, see SAT Command Overview.
In CSM 1.3 and newer, the sat command is automatically available on all the
+Kubernetes control plane. For more information, see SAT in CSM. Older
+versions of CSM do not have the sat command automatically available, and SAT
+must be installed as a separate product.
Describes the SAT Command Line Utility, lists the key commands found in the System Admin Toolkit man pages, and provides +instruction on the SAT Container Environment.
+The primary component of the System Admin Toolkit (SAT) is a command-line utility run from Kubernetes control plane nodes
+(ncn-m nodes).
It is designed to assist administrators with common tasks, such as troubleshooting and querying information about the
+HPE Cray EX System and its components, system boot and shutdown, and replacing hardware components. There are
+similarities between SAT commands and xt commands used on the Cray XC platform.
The top-level SAT man page describes the toolkit, documents the global options affecting all subcommands, documents +configuration file options, and references the man page for each subcommand. SAT consists of many subcommands that each +have their own set of options.
+The sat command-line utility runs in a container using Podman, a daemonless container runtime. SAT runs on
+Kubernetes control plane nodes. A few important points about the SAT container environment include the following:
sat or sat bash always launches a container.There are two ways to run sat.
sat bash, followed by a sat command.sat command directly on a Kubernetes control plane node.In both of these cases, a container is launched in the background to execute the command. The first option, running
+sat bash first, gives an interactive shell, at which point sat commands can be run. In the second option, the
+container is launched, executes the command, and upon the command’s completion the container exits. The following two
+examples show the same action, checking the system status, using both modes.
(ncn-m001#) Here is an example using interactive mode:
sat bash
+((CONTAINER_ID) sat-container#) Example sat command after a container is launched:
sat status
+(ncn-m001#) Here is an example using non-interactive mode:
sat status
+Running sat using the interactive command prompt gives the ability to read and write local files on ephemeral
+container storage. If multiple sat commands are being run in succession, use sat bash to launch the
+container beforehand. This will save time because the container does not need to be launched for each sat command.
The non-interactive mode is useful if calling sat with a script, or when running a single sat command as a part of
+several steps that need to be executed from a management NCN.
To view a sat man page from a Kubernetes control plane node, use sat-man on the manager node.
(ncn-m001#) Here is an example:
sat-man status
+A man page describing the SAT container environment is available on the Kubernetes control plane nodes, which can be viewed
+either with man sat or man sat-podman from the manager node.
(ncn-m001#) Here are examples:
man sat
+man sat-podman
+The host name in a command prompt indicates where the command must be run. The +user account that must run the command is also indicated in the prompt.
+root or super-user account always has host name in the prompt and the
+# character at the end of the prompt.account@hostname>. A non-privileged
+account is referred to as user.# character at the end of the prompt.| Command Prompt | +Meaning | +
|---|---|
ncn-m001# |
+Run the command as root on the specific Kubernetes control plane server which has this hostname (ncn-m001 in this example). (Non-interactive) |
+
user@hostname> |
+Run the command as any non-root user on the specified hostname. (Non-interactive) |
+
(venv) user@hostname> |
+Run the command as any non-root user within a Python virtual environment on the specified hostname. (Non-interactive) |
+
(CONTAINER_ID) sat-container# |
+Run the command inside the SAT container environment by first running sat bash. (Interactive) |
+
These command prompts should be inserted into text before the fenced code block +instead of inside of it. This is a change from the documentation of SAT 2.5 and +earlier. Here is an example of the new use of the command prompt:
+(ncn-m001#) Example first step.
yes >/dev/null
+In CSM 1.3 and newer, the sat command is automatically available on the Kubernetes control plane, but it is still possible
+to install SAT as a separate product stream. Any version of SAT installed as a separate product stream overrides the
+sat command available in CSM. Installing the SAT product stream allows additional supporting components to be added:
An entry for SAT in the cray-product-catalog Kubernetes ConfigMap is only created by installing the SAT product
+stream. Otherwise, there will be no entry for this version of SAT in the output of sat showrev.
The sat-install-utility container image is only available with the full SAT product stream. This container image
+provides uninstall and downgrade functionality when used with the prodmgr command. (In SAT 2.3 and older, SAT was
+only available to install as a separate product stream. Because these versions were packaged with
+sat-install-utility, it is still possible to uninstall these versions of SAT.)
The docs-sat RPM package is only available with the full SAT product stream.
The sat-config-management git repository in Gitea (VCS) and thus the SAT layer of NCN CFS configuration is
+only available with the full SAT product stream.
If the SAT product stream is not installed, there will be no configuration content for SAT in VCS. Therefore, CFS
+configurations that apply to management NCNs (for example, management-23.5.0) should not include a SAT layer.
The SAT configuration layer modifies the permissions of files left over from prior installations of SAT, so that the
+Keycloak username that authenticates to the API gateway cannot be read by users other than root. Specifically,
+it does the following:
Modifies the sat.toml configuration file which contains the username so that it is only readable by root.
Modifies the /root/.config/sat/tokens directory so that the directory is only readable by root. This is needed
+because the names of the files within the tokens directory contain the username.
Regardless of the SAT configuration being applied, passwords and the contents of the tokens are never readable by other +users. These permission changes only apply to files created by previous installations of SAT. In the current version of +SAT all files and directories are created with the appropriate permissions.
+ + + + + +View the System Admin Toolkit (SAT) documentation both online and +offline by using the information in this section.
+The SAT documentation can be found online in HTML form at the following link: +SAT Documentation. The navigation pane +on the left of the HTML page orders topics alphabetically. Navigate an +individual topic’s headings by using the Headings icon at the top of the +page, as shown in the following images.
+
The documentation can also be viewed online in GitHub by navigating to the
+docs/ subdirectory of the
+docs-sat repository.
+Navigate an individual topic’s headings with a similar
+Headings icon at the top of the page, as shown in the following images.
The SAT documentation is available offline as markdown, which can be
+viewed with a markdown viewer or with a text editor. The offline
+documentation is available in the docs/ directory of the SAT release
+distribution as well as in RPM package format. The RPM package is
+installed as a part of the Ansible plays launched by the Configuration
+Framework Service (CFS). Its files are installed to /usr/share/doc/sat.
SAT can optionally be installed and configured on an external system to interact +with CSM over the CAN.
+Most SAT subcommands work by accessing APIs which are reachable via the CAN. +However, certain SAT commands depend on host-based functionality on the +management NCNs and will not work from an external system. This includes the +following:
+platform-services and ncn-power stages of sat bootsys--local option of sat showrevInstalling SAT on an external system is not an officially supported configuration. +These instructions are provided “as-is” with the hope that they can be useful for +users who desire additional flexibility.
+Certain additional steps may need to be taken to install and configure SAT +depending on the configuration of the external system in use. These additional +steps may include provisioning virtual machines, installing packages, or +configuring TLS certificates, and these steps are outside the scope of this +documentation. This section covers only the steps needed to configure SAT to +use externally-accessible API endpoints exposed by CSM.
+kubectl, openssh, git, and curl are installed on the external system.(user@hostname>) Create a Python virtual environment.
SAT_VENV_PATH="$(pwd)/venv"
+python3 -m venv ${SAT_VENV_PATH}
+. ${SAT_VENV_PATH}/bin/activate
+((venv) user@hostname>) Clone the SAT source code.
To use SAT version 3.21, this example clones the release/3.21 branch of
+Cray-HPE/sat.
git clone --branch=release/3.21 https://github.com/Cray-HPE/sat.git
+Set up the SAT CSM Python dependencies to be installed from their source code.
+SAT CSM Python dependency packages are not currently distributed publicly as
+source packages or binary distributions. They must be installed from
+their source code hosted on GitHub. Also, to install the cray-product-catalog
+Python package, first clone it locally. Use the following steps to
+modify the SAT CSM Python dependencies so they can be installed from their source
+code.
((venv) user@hostname>) Clone the source code for cray-product-catalog.
git clone --branch v1.6.0 https://github.com/Cray-HPE/cray-product-catalog
+((venv) user@hostname>) In the cray-product-catalog directory, create a file named .version
+that contains the version of cray-product-catalog.
echo 1.6.0 > cray-product-catalog/.version
+((venv) user@hostname>) Open the “locked” requirements file in a text editor.
vim sat/requirements.lock.txt
+Update the line containing cray-product-catalog so that it reflects the
+local path to cray-product-catalog.
It should read as follows:
+./cray-product-catalog
+For versions of SAT newer than 3.19, change the line containing csm-api-client
+to read as follows.
csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+(Optional) ((venv) user@hostname>) Confirm that requirements.lock.txt is modified as expected.
grep -E 'cray-product-catalog|csm-api-client' sat/requirements.lock.txt
+Example output:
+./cray-product-catalog
+csm-api-client@git+https://github.com/Cray-HPE/python-csm-api-client@release/1.1
+Note: For versions newer than 3.19, the output will show both
+cray-product-catalog and csm-api-client. For version 3.19 and older,
+the output will only show cray-product-catalog.
((venv) user@hostname>) Install the modified SAT dependencies.
pip install -r sat/requirements.lock.txt
+((venv) user@hostname>) Install the SAT Python package.
pip install ./sat
+(Optional) ((venv) user@hostname>) Add the sat virtual environment to the user’s PATH environment
+variable.
If a shell other than bash is in use, replace ~/.bash_profile with the
+appropriate profile path.
If the virtual environment is not added to the user’s PATH environment
+variable, then source ${SAT_VENV_PATH}/bin/activate will need to be run before
+running any SAT commands.
deactivate
+echo export PATH=\"${SAT_VENV_PATH}/bin:${PATH}\" >> ~/.bash_profile
+source ~/.bash_profile
+(user@hostname>) Copy the file /etc/kubernetes/admin.conf from ncn-m001 to ~/.kube/config
+on the external system.
Note that this file contains credentials to authenticate against the Kubernetes +API as the administrative user, so it should be treated as sensitive.
+mkdir -p ~/.kube
+scp ncn-m001:/etc/kubernetes/admin.conf ~/.kube/config\
+Example output:
+admin.conf 100% 5566 3.0MB/s 00:00
+(user@hostname>) Find the CAN IP address on ncn-m001 to determine the
+corresponding kubernetes hostname.
On CSM 1.2 and newer, query the IP address of the bond0.cmn0
+interface.
ssh ncn-m001 ip addr show bond0.cmn0
+Example output:
+13: bond0.cmn0@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.11/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+On CSM versions prior to 1.2, query the IP address of the vlan007 interface.
ssh ncn-m001 ip addr show vlan007
+Example output:
+13: vlan007@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+link/ether b8:59:9f:1d:d9:0e brd ff:ff:ff:ff:ff:ff
+inet 10.102.1.10/24 brd 10.102.1.255 scope global vlan007
+ valid_lft forever preferred_lft forever
+inet6 fe80::ba59:9fff:fe1d:d90e/64 scope link
+ valid_lft forever preferred_lft forever
+(user@hostname>) Set the IP_ADDRESS variable to the value found in the
+previous step.
IP_ADDRESS=10.102.1.11
+(user@hostname>) Add an entry to /etc/hosts mapping the IP address to
+the hostname kubernetes.
echo "${IP_ADDRESS} kubernetes" | sudo tee -a /etc/hosts
+10.102.1.11 kubernetes
+(user@hostname>) Modify ~/.kube/config to set the cluster server address.
The value of the server key for the kubernetes cluster under the clusters
+section should be set to https://kubernetes:6443.
---
+clusters:
+- cluster:
+ certificate-authority-data: REDACTED
+ server: https://kubernetes:6443
+ name: kubernetes
+...
+(user@hostname>) Confirm that kubectl can access the CSM Kubernetes cluster.
kubectl get nodes
+Example output:
+NAME STATUS ROLES AGE VERSION
+ncn-m001 Ready master 135d v1.19.9
+ncn-m002 Ready master 136d v1.19.9
+ncn-m003 Ready master 136d v1.19.9
+ncn-w001 Ready <none> 136d v1.19.9
+ncn-w002 Ready <none> 136d v1.19.9
+ncn-w003 Ready <none> 136d v1.19.9
+(user@hostname>) Use sat init to create a configuration file for SAT.
sat init
+Example output:
+INFO: Configuration file "/home/user/.config/sat/sat.toml" generated.
+(user@hostname>) Copy the platform CA certificates from the management NCN
+and configure the certificates for use with SAT.
If a shell other than bash is in use, replace ~/.bash_profile with the
+appropriate profile path.
scp ncn-m001:/etc/pki/trust/anchors/platform-ca-certs.crt .
+echo export REQUESTS_CA_BUNDLE=\"$(realpath platform-ca-certs.crt)\" >> ~/.bash_profile
+source ~/.bash_profile
+Edit the SAT configuration file to set the API and S3 hostnames.
+Externally available API endpoints are given domain names in PowerDNS, so the
+endpoints in the configuration file should each be set to the format
+subdomain.system-name.site-domain. Here system-name and site-domain are
+replaced with the values specified during csi config init, and subdomain
+is the DNS name for the externally available service. For more information,
+refer to Externally Exposed Services in the Cray System Management
+Documentation.
The API gateway has the subdomain api, and S3 has the subdomain s3. The
+S3 endpoint runs on port 8080. The following options should be set in the
+SAT configuration file.
[api_gateway]
+host = "api.system-name.site-domain"
+
+[s3]
+endpoint = "http://s3.system-name.site-domain:8080"
+Edit the SAT configuration file to specify the Keycloak user who will be +accessing the REST API.
+[api_gateway]
+username = "user"
+(user@hostname>) Run sat auth, and enter the password when prompted.
The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin.
sat auth
+Example output:
+Password for user:
+Succeeded!
+For more information on authentication types and authentication credentials, +see SAT Command Authentication. +For more information on Keycloak accounts and changing Role Mappings, +refer to both Configure Keycloak Account and Create Internal User +Accounts in the Keycloak Shasta Realm in the Cray System Management +Documentation.
+(user@hostname>) Ensure the files are readable only by the current user.
touch ~/.config/sat/s3_access_key \
+ ~/.config/sat/s3_secret_key
+chmod 600 ~/.config/sat/s3_access_key \
+ ~/.config/sat/s3_secret_key
+(user@hostname>) Write the credentials to local files using kubectl.
Generate S3 credentials and write them to a local file so the SAT user can
+access S3 storage. In order to use the SAT S3 bucket, the user must generate
+the S3 access key and secret keys and write them to a local file. SAT uses
+S3 storage for several purposes, most importantly to store the site-specific
+information set with sat setrev.
kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ ~/.config/sat/s3_access_key
+kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ ~/.config/sat/s3_secret_key
+IMPORTANT: Starting in CSM 1.6.0, SAT is fully included in CSM. There is no longer a separate SAT +product stream to install. SAT 2.6 releases, which accompanied CSM 1.5, are the last releases of +SAT as a separate product.
+Similarly, the SAT documentation moved to be fully included within the CSM documentation. Starting in +CSM 1.6.0, find information on SAT in the +System Admin Toolkit (SAT) section +of the Cray System Management Documentation.
+The Install and Upgrade Framework (IUF) provides commands which install, +upgrade, and deploy products on systems managed by CSM. IUF capabilities are +described in detail in the IUF +section of the +Cray System Management Documentation. +The initial install and upgrade workflows described in the +HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM +(S-8052) detail when and how to use +IUF with a new release of SAT or any other HPE Cray EX product.
+This document does not replicate install, upgrade, or deployment procedures +detailed in the Cray System Management +Documentation. This document provides +details regarding software and configuration content specific to SAT which is +needed when installing, upgrading, or deploying a SAT release. The Cray +System Management Documentation will +indicate when sections of this document should be referred to for detailed +information.
+IUF will perform the following tasks for a release of SAT.
+deliver-product stage:
+update-vcs-config stage:
+update-cfs-config stage:
+prepare-images stage:
+management-nodes-rollout stage:
+IUF uses a variety of CSM and SAT tools when performing these tasks. The IUF +section of the +Cray System Management Documentation +describes how to use these tools directly if it is desirable to use them +instead of IUF.
+This section describes SAT details that an administrator must be aware of +before running IUF stages. Entries are prefixed with Information if no +administrative action is required or Action if an administrator needs +to perform tasks outside of IUF.
+Information: This stage is only run if a VCS working branch is specified for +SAT. By default, SAT does not create or specify a VCS working branch.
+Information: This stage only applies to the management configuration and +not to the managed configuration.
+Information: This stage only applies to management images and not to +managed images.
+After installing SAT with IUF, complete the following SAT configuration +procedures before using SAT:
+...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being installed.To run SAT commands on the manager NCNs, first set up authentication +to the API gateway. For more information on authentication types and +authentication credentials, see SAT Command +Authentication.
+The admin account used to authenticate with sat auth must be enabled in
+Keycloak and must have its assigned role set to admin. For more information
+on Keycloak accounts and changing Role Mappings, refer to both Configure Keycloak
+Account and Create Internal User Accounts in the Keycloak Shasta Realm in
+the Cray System Management Documentation.
sat CLI has been installed following the IUF
+section of the
+Cray System Management Documentation.The following is the procedure to globally configure the username used by SAT and +authenticate to the API gateway.
+(ncn-m001#) Generate a default SAT configuration file if one does not exist.
sat init
+Example output:
+Configuration file "/root/.config/sat/sat.toml" generated.
+Note: If the configuration file already exists, it will print out the +following error.
+ERROR: Configuration file "/root/.config/sat/sat.toml" already exists.
+Not generating configuration file.
+Edit ~/.config/sat/sat.toml and set the username option in the api_gateway
+section of the configuration file.
username = "crayadmin"
+(ncn-m001#) Run sat auth. Enter the password when prompted.
sat auth
+Example output:
+Password for crayadmin:
+Succeeded!
+(ncn-m001#) Other sat commands are now authenticated to make requests to the API gateway.
sat status
+Generate S3 credentials and write them to a local file so the SAT user can access +S3 storage. In order to use the SAT S3 bucket, the System Administrator must +generate the S3 access key and secret keys and write them to a local file. This +must be done on every Kubernetes control plane node where SAT commands are run.
+SAT uses S3 storage for several purposes, most importantly to store the
+site-specific information set with sat setrev (see Set System Revision
+Information).
(ncn-m001#) Ensure the files are readable only by root.
touch /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+chmod 600 /root/.config/sat/s3_access_key \
+ /root/.config/sat/s3_secret_key
+(ncn-m001#) Write the credentials to local files using kubectl.
kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.access_key}' | base64 -d > \
+ /root/.config/sat/s3_access_key
+kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.secret_key}' | base64 -d > \
+ /root/.config/sat/s3_secret_key
+Verify the S3 endpoint specified in the SAT configuration file is correct.
+(ncn-m001#) Get the SAT configuration file’s endpoint value.
Note: If the command’s output is commented out, indicated by an initial #
+character, the SAT configuration will take the default value – "https://rgw-vip.nmn".
grep endpoint ~/.config/sat/sat.toml
+Example output:
+# endpoint = "https://rgw-vip.nmn"
+(ncn-m001#) Get the sat-s3-credentials secret’s endpoint value.
kubectl get secret sat-s3-credentials -o json -o \
+ jsonpath='{.data.s3_endpoint}' | base64 -d | xargs
+Example output:
+https://rgw-vip.nmn
+Compare the two endpoint values.
+If the values differ, change the SAT configuration file’s endpoint value to +match the secret’s.
+(ncn-m001#) Copy SAT configurations to each manager node on the system.
for i in ncn-m002 ncn-m003; do echo $i; ssh ${i} \
+ mkdir -p /root/.config/sat; \
+ scp -pr /root/.config/sat ${i}:/root/.config; done
+Note: Depending on how many manager nodes are on the system, the list of
+manager nodes may be different. This example assumes three manager nodes, where
+the configuration files must be copied from ncn-m001 to ncn-m002 and
+ncn-m003. Therefore, the list of hosts above is ncn-m002 and ncn-m003.
If installing SAT on a multi-tenant system, the tenant name can be configured +at this point. For more information, see Configure multi-tenancy.
+HPE service representatives use system revision information data to identify +systems in support cases.
+(ncn-m001#) Set System Revision Information.
Run sat setrev and follow the prompts to set the following site-specific values:
Tip: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. In other words, “System type” is EX-1C.
+sat setrev
+Example output:
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Verify System Revision Information.
+(ncn-m001#) Run sat showrev and verify the output shown in the “System Revision Information table.”
sat showrev
+Example table output:
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+SAT 2.2.16 was released on February 25th, 2022.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container image and Helm chartIt also added the following new components:
+sat-install-utility container imagecfs-config-util container imageThe following sections detail the changes in this release.
+sat Command Unavailable in sat bash ShellAfter launching a shell within the SAT container with sat bash, the sat
+command will not be found.
((CONTAINER_ID) sat-container#) Here is an example output after running sat status:
bash: sat: command not found
+((CONTAINER_ID) sat-container#) This can be resolved temporarily in one of two ways. /sat/venv/bin/ may be
+prepended to the $PATH environment variable:
export PATH=/sat/venv/bin:$PATH
+sat status
+((CONTAINER_ID) sat-container#) Another option is to source the file /sat/venv/bin/activate:
source /sat/venv/bin/activate
+sat status
+sat bash ShellAfter launching a shell within the SAT container with sat bash, tab completion
+for sat commands does not work.
((CONTAINER_ID) sat-container#) This can be resolved temporarily by sourcing the file
+/etc/bash_completion.d/sat-completion.bash:
source /etc/bash_completion.d/sat-completion.bash
+sat in Root Directorysat commands will not work if the current directory is /.
(ncn-m001#) Here is an example output after running sat --help:
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: open /dev/console: operation not permitted: OCI runtime permission denied error
+To resolve, run sat in another directory.
sat in Configuration Directorysat commands will not work if the current directory is ~/.config/sat.
(ncn-m001#) Here is an example output after running sat --help:
Error: /root/.config/sat: duplicate mount destination
+To resolve, run sat in another directory.
sat Commandssat bootprep automates the creation of CFS configurations, the build and
+customization of IMS images, and the creation of BOS session templates. For
+more information, see SAT Bootprep.sat slscheck performs a check for consistency between the System Layout
+Service (SLS) and the Hardware State Manager (HSM).sat bmccreds provides a simple interface for interacting with the System
+Configuration Service (SCSD) to set BMC Redfish credentials.sat hwhist displays hardware component history by XName (location) or by
+its Field-Replaceable Unit ID (FRUID). This command queries the Hardware
+State Manager (HSM) API to obtain this information. Since the sat hwhist
+command supports querying for the history of a component by its FRUID, the
+FRUID of components has been added to the output of sat hwinv.The following automation has been added to the install script, install.sh:
sat-config-import Kubernetes job, which is
+started when the sat-cfs-install Helm chart is deployed.ncn-personalization).The SAT product uploads additional information to the cray-product-catalog
+Kubernetes ConfigMap detailing the components it provides, including container
+(Docker) images, Helm charts, RPMs, and package repositories.
This information is used to support uninstall and downgrade of SAT product +versions moving forward.
+Beginning with the 2.2 release, SAT now provides partial support for the +uninstall and downgrade of the SAT product stream.
+For more information, see +Uninstall: Remove a Version of SAT and +Downgrade: Switch Between SAT Versions.
+sat statusA Subrole column has been added to the output of sat status. This allows
+easy differentiation between master, worker, and storage nodes in the
+management role, for example.
Hostname information from SLS has been added to sat status output.
Support for JSON-formatted output has been added to commands which currently
+support the --format option, such as hwinv, status, and showrev.
Many usability improvements have been made to multiple sat commands,
+mostly related to filtering command output. The following are some highlights:
--fields option to display only specific fields for subcommands which
+display tabular reports.--filter queries
+so that the first match is used, similar to --sort-by.--filter, --fields, and --reverse for summaries
+displayed by sat hwinv.sat hwinv.The default log level for stderr has been changed from “WARNING” to “INFO”. For
+more information, see Update SAT Logging.
With the command-line options --loglevel-stderr and --loglevel-file, the log
+level can now be configured separately for stderr and the log file.
The existing --loglevel option is now an alias for the --loglevel-stderr
+option.
The Podman wrapper script is the script installed at /usr/bin/sat on the
+master management NCNs by the cray-sat-podman RPM that runs the cray-sat
+container in podman. The following subsections detail improvements that were
+made to the wrapper script in this release.
cray-sat ContainerThe Podman wrapper script that launches the cray-sat container with podman
+has been modified to mount the user’s current directory and home directory into
+the cray-sat container to provide access to local files in the container.
The man page for the Podman wrapper script, which is accessed by typing man sat on a master management NCN, has been improved to document the following:
Fixed issues with redirecting stdout and stderr, and piping output to
+commands, such as awk, less, and more.
A new sat option has been added to configure the HTTP timeout length for
+requests to the API gateway. For more information, refer to sat-man sat.
sat bootsys ImprovementsMany improvements and fixes have been made to sat bootsys. The following are
+some highlights:
--excluded-ncns option, which can be used to omit NCNs
+from the platform-services and ncn-power stages in case they are
+inaccessible.sat bootsys shutdown now prompt the user to
+continue before proceeding. A new option, --disruptive, will bypass this.platform-services stage of sat bootsys boot.sat xname2nid Improvementssat xname2nid can now recursively expand slot, chassis, and cabinet XNames to
+a list of NIDs in those locations.
A new --format option has been added to sat xname2nid. It sets the output
+format to either “range” (the default) or “NID”. The “range” format displays NIDs
+in a compressed range format suitable for use with a workload manager like Slurm.
v2 HSM APIThe commands which interact with HSM (for example, sat status and sat hwinv)
+now use the v2 HSM API.
sat diag Limited to HSN Switchessat diag will now only operate against HSN switches by default. These are the
+only controllers that support running diagnostics with HMJTD.
sat showrev EnhancementsA column has been added to the output of sat showrev that indicates whether a
+product version is “active”. The definition of “active” varies across products,
+and not all products may set an “active” version.
For SAT, the active version is the one with its hosted-type package repository
+in Nexus set as the member of the group-type package repository in Nexus,
+meaning that it will be used when installing the cray-sat-podman RPM.
cray-sat Container Image Size ReductionThe size of the cray-sat container image has been approximately cut in half by
+leveraging multi-stage builds. This also improved the repeatability of the unit
+tests by running them in the container.
Minor bug fixes were made in cray-sat and in cray-sat-podman. For full
+change lists, refer to each repository’s CHANGELOG.md file.
The 2.3.4 version of the SAT product includes:
+sat python package and CLIsat-podman wrapper scriptsat-cfs-install container imagesat-cfs-install Helm chartsat-install-utility container imagecfs-config-util container imagesat CommandsNone.
+When running sat commands, the current working directory is now mounted in the
+container as /sat/share, and the current working directory within the container
+is also /sat/share.
Files in the current working directory must be specified using relative paths to
+that directory, because the current working directory is always mounted on
+/sat/share. Absolute paths should be avoided, and paths that are outside of
+$HOME or $PWD are never accessible to the container environment.
The home directory is still mounted on the same path inside the container as it +is on the host.
+sat bootsysThe following options were added to sat bootsys.
--bos-limit--recursiveThe --bos-limit option passes a given limit string to a BOS session. The
+--recursive option specifies a slot or other higher-level component in the
+limit string.
sat bootprepThe --delete-ims-jobs option was added to sat bootprep run. It deletes IMS
+jobs after sat bootprep is run. Jobs are no longer deleted by default.
sat statussat status now includes information about nodes’ CFS configuration statuses,
+such as desired configuration, configuration status, and error count.
The output of sat status now splits different component types into different
+report tables.
The following options were added to sat status.
--hsm-fields, --sls-fields, --cfs-fields--bos-templateThe --hsm-fields, --sls-fields, --cfs-fields options limit the output
+columns according to specified CSM services.
The --bos-template option filters the status report according to the specified
+session template’s boot sets.
The following components were modified to be compatible with CSM 1.2.
+sat-cfs-install container image and Helm chartsat-install-utility container imageThe sat-ncn Ansible role provided by sat-cfs-install was modified to enable
+GPG checks on packages while leaving GPG checks disabled on repository metadata.
Updated urllib3 dependency to version 1.26.5 to mitigate CVE-2021-33503 and
+refreshed Python dependency versions.
Minor bug fixes were made in each of the repositories. For full change lists,
+refer to each repository’s CHANGELOG.md file.
The known issues listed under the SAT 2.2 release +were fixed.
+ + + + + +The 2.4.13 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.cfs-config-util container image.Because of installation refactoring efforts, the following two components +are no longer delivered with SAT:
+sat-cfs-install container imagesat-cfs-install Helm chartA version of the cray-sat container image is now included in CSM. For more
+information, see SAT in CSM.
The SAT install.sh script no longer uses a sat-cfs-install Helm chart and
+container image to upload its Ansible content to the sat-config-management
+repository in VCS. Instead, it uses Podman to run the cf-gitea-import container
+directly. Some of the benefits of this change include the following:
cray-sat container image and cray-sat-podman packagecray-sat Container Image and cray-sat-podman PackageIn older SAT releases, the sat wrapper script that was provided by the
+cray-sat-podman package installed on Kubernetes control plane nodes included a
+hard-coded version of the cray-sat container image. As a result, every new
+version of the cray-sat image required a corresponding new version of the
+cray-sat-podman package.
In this release, this tight coupling of the cray-sat-podman package and the
+cray-sat container image was removed. The sat wrapper script provided
+by the cray-sat-podman package now looks for the version of the cray-sat
+container image in the /opt/cray/etc/sat/version file. This file is populated
+with the correct version of the cray-sat container image by the SAT layer of
+the CFS configuration that is applied to management NCNs. If the version file
+does not exist, the wrapper script defaults to the version of the cray-sat
+container image delivered with the latest version of CSM installed on the system.
The steps for performing NCN personalization as part of the SAT installation
+were moved out of the install.sh script and into a new
+update-mgmt-ncn-cfs-config.sh script that is provided in the SAT release
+distribution. The new script provides additional flexibility in how it modifies
+the NCN personalization CFS configuration for SAT. It can modify an existing CFS
+configuration by name, a CFS configuration being built in a JSON file, or an
+existing CFS configuration that applies to certain components.
sat bootprep FeaturesThe following new features were added to the sat bootprep command:
Variable substitutions using Jinja2 templates in certain fields of the
+sat bootprep input file
For more information, see +HPC CSM Software Recipe Variable Substitutions +and Dynamic Variable Substitutions.
+Schema version validation in the sat bootprep input files
For more information, see +Provide a Schema Version.
+Ability to look up images and recipes provided by products
+For more information, see +Define IMS Images.
+The schema of the sat bootprep input files was also changed to support these
+new features:
base key instead of under an ims key. The old ims
+key is deprecated.base.image_ref.
+Going forward, do not use the IMS name of the image on which it depends.image.ims.name, image.ims.id, or image.image_ref. Specifying a string
+value directly under the image key is deprecated.For more information on defining IMS images and BOS session templates in the
+sat bootprep input file, see Define IMS Images
+and Define BOS Session Templates.
sat swapThe sat swap command was updated to support swapping compute and UAN blades
+with sat swap blade. This functionality is described in the following processes
+of the Cray System Management Documentation:
v2A new v2 version of the Boot Orchestration Service (BOS) is available in CSM
+1.3.0. SAT has added support for BOS v2. This impacts the following commands
+that interact with BOS:
sat bootprepsat bootsyssat statusBy default, SAT uses BOS v1. To change the default to a different BOS version,
+see Change the BOS Version.
sat statusWhen using BOS v2, sat status outputs additional fields. These fields show
+the most recent BOS session, session template, booted image, and boot status for
+each node. An additional --bos-fields option was added to limit the output of
+sat status to these fields. The fields are not displayed when using BOS v1.
This is the first release of SAT built from open source code repositories. +As a result, build infrastructure was changed to use an external Jenkins instance, +and artifacts are now published to an external Artifactory instance. These +changes should not impact the functionality of the SAT product in any way.
+paramiko Python package version was updated from 2.9.2 to 2.10.1 to
+mitigate CVE-2022-24302.oauthlib Python package version was updated from 3.2.0 to 3.2.1 to
+mitigate CVE-2022-36087.SAT stores information used to authenticate to the API gateway with Keycloak.
+Token files are stored in the ~/.config/sat/tokens/ directory. Those files
+have always had permissions appropriately set to restrict them to be readable
+only by the user.
Keycloak usernames used to authenticate to the API gateway are stored in the
+SAT configuration file at /.config/sat/sat.toml. Keycloak usernames are also
+used in the file names of tokens stored in /.config/sat/tokens. As an
+additional security measure, SAT now restricts the permissions of the SAT
+configuration file to be readable and writable only by the user. It also
+restricts the tokens directory and the entire SAT configuration directory
+~/.config/sat to be accessible only by the user. This prevents other users on
+the system from viewing Keycloak usernames used to authenticate to the API
+gateway.
sat init did not print a message confirming a new
+configuration file was created.sat showrev exited with a traceback if the file
+/opt/cray/etc/site_info.yaml existed but was empty. This could occur if the
+user exited sat setrev with Ctrl-C.sat bootsys man page, and added a
+description of the command stages.The 2.5.17 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.cfs-config-util container image.sat Commandssat jobstat allows access to application and job data through the command
+line. It provides a table summarizing information for all jobs on the system.
sat bootprepA list-vars subcommand was added to sat bootprep.
It lists the variables available for use in bootprep input files at runtime.
+A --limit option was added to sat bootprep run.
It restricts the creation of CFS configurations, IMS images, and BOS session +templates into separate stages. For more information, see +Limit SAT Bootprep Run into Stages.
+sat bootprep now prompts individually for each CFS configuration that
+already exists.
sat bootprep can now filter images provided by a product by using a prefix.
This is useful when specifying the base of an image in a bootprep input +file. For more information, see +Define IMS Images.
+To support product names with hyphens, sat bootprep now converts hyphens to
+underscores within variables.
For more information, see +Hyphens in HPC CSM Software Recipe Variables.
+In sat bootprep input files, the value of the playbook property of CFS
+configuration layers can now be rendered with Jinja2 templates.
For more information, see +Values Supporting Jinja2 Template Rendering.
+Output was added to sat bootprep run that summarizes the CFS configurations,
+IMS images, and BOS session templates created.
For more information, see +Summary of SAT Bootprep Results.
+Improvements were made to the sat bootprep output when CFS configuration
+and BOS session templates are created.
sat bootsysreboot subcommand was added to sat bootsys. It uses BOS to reboot
+nodes in the bos-operations stage.--staged-session option was added to sat bootsys. It can be used to
+create staged BOS sessions. For more information, refer to Staging Changes
+with BOS in the Cray System Management Documentation.sat Commandsprodmgr, a version is no longer set as
+“active” in the product catalog. The “active” field was also removed from the
+output of sat showrev.sat status when using BOS
+version two.The new Install and Upgrade Framework (IUF) provides commands which install,
+upgrade, and deploy products with the help of sat bootprep on HPE Cray EX
+systems managed by Cray System Management (CSM). IUF capabilities are described
+in detail in the IUF section
+of the Cray System Management Documentation.
+The initial install and upgrade workflows described in the
+HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM
+(S-8052) detail when and how to use
+IUF with a new release of SAT or any other HPE Cray EX product.
Because IUF now handles NCN personalization, information about this process was +removed from the SAT documentation. Other sections in the documentation were +also revised to support the new Install and Upgrade Framework. For example, the +SAT Installation and SAT Upgrade sections of this +guide now provide details on software and configuration content specific to SAT. +The Cray System Management Documentation +will indicate when these sections should be referred to for detailed information.
+For more information on the relationship between sat bootprep and IUF, see
+SAT and IUF.
By default, SAT now uses version two of the Boot Orchestration Service (BOS).
+This change to BOS v2 impacts the following commands that interact with BOS:
sat bootprepsat bootsyssat statusTo change the default to a different BOS version, see +Change the BOS Version.
+sat python package and CLI from
+2021.10.8 to 2022.12.7 to resolve CVE-2022-23491.sat-install-utility container image
+from 2021.5.30 to 2022.12.7 to resolve CVE-2022-23491.sat init from creating a configuration file in
+the current directory when not prefixed with ./.sat status failed with a traceback when using BOS
+version two and reported components whose most recent image did not exist.sat container could contain a different
+version of kubectl than the version found in CSM.sat bootprep and
+sat swap blade.The 2.6.14 version of the SAT product includes:
+sat python package and CLI.sat-podman wrapper script.sat-install-utility container image.sat CommandsNo new sat commands were added in SAT 2.6.
sat bootsysFunctionality was added to the platform-services and cabinet-power
+stages of sat bootsys boot. This allows SAT to automatically recreate
+Kubernetes CronJobs that may have become stuck during shutdown, boot, or
+reboot.
sat bootsys boot more reliably determines if the hms-discovery CronJob
+was scheduled during the cabinet-power stage.
SAT now uses the BatchV1 Kubernetes API to manipulate CronJobs instead of the +BatchV1Beta1 API.
+sat bootsys now logs the ID of all BOS sessions when performing BOS
+operations. A warning is logged for any BOS sessions with failed
+components.
Support for the Compute Rolling Upgrade Service (CRUS) has been removed,
+and the sat bootsys command will no longer interact with CRUS.
The bos-operations stage of sat bootsys no longer checks whether BOS
+session templates need any operations to be performed before creating a BOS
+session. BOS instead determines whether the session will need to boot or
+shut down any nodes to reach the desired state.
sat bootprepWildcard matching was added for images in sat bootprep input files. Use
+wildcards similar to how prefix filters were used in older versions of SAT.
+For more information, see Define IMS Images.
Support for multiple architectures was added to sat bootprep. It is now
+possible to filter base IMS images and recipes from products based on their
+target architecture. This support also allows specifying target architectures
+in boot sets of BOS session templates. For more information, see
+Filter Base Images or Recipes from a Product
+and
+Define BOS Session Templates.
When specifying a base image or recipe from a product, sat bootprep
+can combine multiple image or recipe filters. When specifying multiple
+filters, the unique base image or recipe that satisfies all of the given
+filters is selected. An error occurs if either no images or recipes match the
+given filters or if more than one image or recipe matches the given filters.
In CFS configuration layers, support was added for the new imsRequireDkms
+field under the specialParameters section. CFS configurations in bootprep
+input files can specify an ims_require_dkms field in a new, optional
+special_parameters section for each layer.
The SAT Kibana and Grafana dashboards were moved to the System Monitoring +Application (SMA) beside other dashboards. For more information on how to +view these dashboards going forward, see the HPE Cray EX System Monitoring +Application Administration Guide (S-8029).
+Add the new s3.cert_verify option to the SAT configuration file to
+control whether certificate verification is performed when accessing S3.
Log messages spanning multiple lines now print the log level on each line +instead of only at the beginning of the message.
+When certificate verification is disabled for CSM API requests, only a single +warning now prints at the beginning of SAT’s invocation instead of for +each request.
+sat swap blade more reliably determines if the hms-discovery CronJob was
+scheduled when enabling a blade following a hardware swap.
sat swap blade will use the BatchV1 Kubernetes API to manipulate CronJobs,
+instead of the BatchV1Beta1 API as previously.
Command prompts in this guide are now inserted into text before the +fenced code block instead of inside of it. This is a change from the +documentation of SAT 2.5 and earlier. In addition, two new command prompts +were added for better clarity. For more information, see +Command Prompt Conventions in SAT.
+SAT 2.6 supports supplying tenant information to CSM services in order to allow +tenant admins to use SAT within their tenant. For more information, see +Configure multi-tenancy.
+Updated the version of cryptography from 36.0.1 to 41.0.0 to resolve +CVE-2023-2650.
+Updated the version of requests from 2.27.1 to 2.31.0 to resolve +CVE-2023-32681.
+Updated the version of curl/libcurl from 7.80.0-r6 to 8.1.2-r0 to address +CVE-2023-27536.
+Improved extreme slowness in the platform-services stage of
+sat bootsys shutdown in cases where a large known_hosts file is used on
+the host where SAT is running.
Fixed a bug that caused the wrong container name to be logged when CFS +configuration sessions failed on newer CSM systems.
+Shasta v1.3.2 included version 2.4.0 of the sat python package and CLI.
The following sections detail the changes in this release.
+sat swap Command for Switch and Cable ReplacementThe sat switch command which supported operations for replacing a switch has
+been deprecated and replaced with the sat swap command, which now supports
+replacing a switch OR cable.
The sat swap switch command is equivalent to sat switch. The sat switch
+command will be removed in a future release.
sat bootsys CommandThe sat bootsys command now has multiple stages for both the boot and
+shutdown actions. Please refer to the “System Power On Procedures” and “System
+Power Off Procedures” sections of the Cray Shasta Administration Guide (S-8001)
+for more details on using this command in the context of a full system power off
+and power on.
Shasta v1.3 included version 2.2.3 of the sat python package and CLI.
This version of the sat CLI contained the following commands:
authbootsyscablecheckdiagfirmwarehwinvhwmatchk8slinkhealthsensorssetrevshowrevstatusswapswitchFor more information on each of these commands, see the +SAT Command Overview and the table +of commands in the SAT Command Authentication +section of this document.
+ + + + + +We released version 2.0.4 of the SAT product in Shasta v1.4.1.
+This version of the SAT product included:
+sat python package and CLI.sat-podman wrapper script.The following sections detail the changes in this release.
+Two new commands were added to translate between NIDs and XNames:
+sat nid2xnamesat xname2nidThese commands perform this translation by making requests to the Hardware +State Manager (HSM) API.
+sat swap where creating the offline port policy failed.sat bootsys shutdown --stage bos-operations to no longer forcefully
+power off all compute nodes and application nodes using CAPMC when BOS
+sessions complete or time out.sat bootsys boot --stage cabinet-power.In Shasta v1.4, SAT became an independent product, which meant we began to +designate a version number for the entire SAT product. We released version +2.0.3 of the SAT product in Shasta v1.4.
+This version of the SAT product included the following components:
+sat python package and CLIIt also added the following new component:
+sat-podman wrapper scriptThe following sections detail the changes in this release.
+SAT is now packaged and released as an independent product. The product
+deliverable is called a “release distribution”. The release distribution is a
+gzipped tar file containing an install script. This install script loads the
+cray/cray-sat container image into the Docker registry in Nexus and loads the
+cray-sat-podman RPM into a package repository in Nexus.
In this release, the cray-sat-podman package is still installed in the master
+and worker NCN images provided by CSM. This is changed in SAT 2.1.16 released in
+Shasta v1.5.
The sat command now runs in a container under Podman. The sat executable is
+now installed on all nodes in the Kubernetes cluster (workers and
+control plane nodes). This executable is a wrapper script that starts a SAT container in
+Podman and invokes the sat Python CLI within that container. The admin can run
+individual sat commands directly on the master or worker NCNs as before, or
+they can run sat commands inside the SAT container after using sat bash to
+enter an interactive shell inside the SAT container.
To view man pages for sat commands, the user can run sat-man SAT_COMMAND,
+replacing SAT_COMMAND with the name of the sat command. Alternatively,
+the user can enter the sat container with sat bash and use the man command.
sat init Command and Configuration File Location ChangeThe default location of the SAT configuration file has been changed from /etc/sat.toml
+to ~/.config/sat/sat.toml. A new command, sat init, has been added that
+initializes a configuration file in the new default directory. This better supports
+individual users on the system who want their own configuration files.
~/.config/sat is mounted into the container that runs under Podman, so changes
+are persistent across invocations of the sat container. If desired, an alternate
+configuration directory can be specified with the SAT_CONFIG_DIR environment
+variable.
Additionally, if a configuration file does not yet exist when a user runs a sat
+command, one is generated automatically.
sat hwinvAdditional functionality has been added to sat hwinv including:
--list-node-enclosure-power-supplies
+option.--list-node-accels option.
+The count of node accelerators is also included for each node.--list-node-accel-risers option. The count of node accelerator risers is also
+included for each node.--list-node-hsn-nics option. The count of HSN NICs is also included for each node.Documentation for these new options has been added to the man page for sat hwinv.
sat setrev in S3The sat setrev and sat showrev commands now use S3 to store and obtain site
+information, including system name, site name, serial number, install date, and
+system type. Since the information is stored in S3, it will now be consistent
+regardless of the node on which sat is executed.
As a result of this change, S3 credentials must be configured for SAT. For more +information, see Generate SAT S3 Credentials.
+sat showrevsat showrev now shows product information from the cray-product-catalog
+ConfigMap in Kubernetes.
sat showrevThe output from sat showrev has also been changed in the following ways:
--docker and --packages options were considered misleading and have
+been removed.--local option.sat cablecheckThe sat cablecheck command has been removed. To verify that the system’s Slingshot
+network is cabled correctly, admins should now use the show cables command in the
+Slingshot Topology Tool (STT).
sat swap Command Compatibility with Next-gen Fabric ControllerThe sat swap command was added in Shasta v1.3.2. This command used the Fabric
+Controller API. Shasta v1.4 introduced a new Fabric Manager API and removed the
+Fabric Controller API, so this command has been rewritten to use the new
+backwards-incompatible API. Usage of the command did not change.
sat bootsys FunctionalityMuch of the functionality added to sat bootsys in Shasta v1.3.2 was broken
+by changes introduced in Shasta v1.4, which removed the Ansible inventory
+and playbooks.
The functionality in the platform-services stage of sat bootsys has been
+re-implemented to use python directly instead of Ansible. This resulted in
+a more robust procedure with better logging to the sat log file. Failures
+to stop containers on Kubernetes nodes are handled more gracefully, and
+more information about the containers that failed to stop, including how to
+debug the problem, is included.
Improvements were made to console logging setup for non-compute nodes +(NCNs) when they are shut down and booted.
+The following improvements were made to the bos-operations stage
+of sat bootsys:
--bos-templates, and a corresponding configuration
+file option, bos_templates, were added, and the --cle-bos-template and
+--uan-bos-template options and their corresponding configuration file
+options were deprecated.The following functionality has been removed from sat bootsys:
hsn-bringup stage of sat bootsys boot has been removed due to removal
+of the underlying Ansible playbook.bgp-check stage of sat bootys {boot,shutdown} has been removed. It is
+now a manual procedure.The location of the sat log file has changed from /var/log/cray/sat.log to
+/var/log/cray/sat/sat.log. This change simplifies mounting this file into the
+sat container running under Podman.
We released version 2.1.16 of the SAT product in Shasta v1.5.
+This version of the SAT product included:
+sat python package and CLIsat-podman wrapper scriptIt also added the following new component:
+sat-cfs-install docker image and helm chartThe following sections detail the changes in this release.
+This release further decouples the installation of the SAT product from the CSM
+product. The cray-sat-podman RPM is no longer installed in the management
+non-compute node (NCN) image. Instead, the cray-sat-podman RPM is installed on
+all master management NCNs via an Ansible playbook which is referenced by a
+layer of the CFS configuration that applies to management NCNs. This CFS
+configuration is typically named ncn-personalization.
The SAT product now includes a Docker image and a Helm chart named
+sat-cfs-install. The SAT install script, install.sh, deploys the Helm chart
+with Loftsman. This helm chart deploys a Kubernetes job that imports the
+SAT Ansible content to a git repository in VCS (Gitea) named sat-config-management.
+This repository is referenced by the layer added to the NCN personalization
+CFS configuration.
All commands which used to access Redfish directly have either been removed or +modified to use higher-level service APIs. This includes the following commands:
+sat sensorssat diagsat linkhealthThe sat sensors command has been rewritten to use the SMA telemetry API to
+obtain the latest sensor values. The command’s usage has changed slightly, but
+legacy options work as before, so it is backwards compatible. Additionally, new
+commands have been added.
The sat diag command has been rewritten to use a new service called Fox, which
+is delivered with the CSM-Diags product. The sat diag command now launches
+diagnostics using the Fox service, which launches the corresponding diagnostic
+programs on controllers using the Hardware Management Job and Task Daemon
+(HMJTD) over Redfish. Essentially, Fox serves as a proxy for us to start
+diagnostics over Redfish.
The sat linkhealth command has been removed. Its functionality has been
+replaced by functionality from the Slingshot Topology Tool (STT) in the
+fabric manager pod.
The Redfish username and password command line options and configuration file +options have been removed. For more information, see +Remove Obsolete Configuration File Sections.
+sat setrev and sat showrevsat setrev now collects the following information from the admin, which is then
+displayed by sat showrev:
Additional guidance and validation has been added to each field collected by
+sat setrev. This sets the stage for sdu setup to stop collecting this
+information and instead collect it from sat showrev or its S3 bucket.
sat bootsysThe platform-services stage of the sat bootsys boot command has been
+improved to start inactive Ceph services, unfreeze Ceph, and wait for Ceph
+health in the correct order. The ceph-check stage has been removed as it is no
+longer needed.
The platform-services stage of sat bootsys boot now prompts for confirmation
+of the storage NCN hostnames in addition to the Kubernetes control plane and worker nodes.
sat firmware.cray-sat container image.sat firmware command.This procedure can be used to uninstall a version of SAT.
+prodmgr.prodmgr command is available.(ncn-m001#) Use sat showrev to list versions of SAT.
sat showrev --products --filter product_name=sat
+Example output:
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+-------------------+-----------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+-------------------+-----------------------+
+| sat | 2.3.3 | - | - |
+| sat | 2.2.10 | - | - |
++--------------+-----------------+-------------------+-----------------------+
+(ncn-m001#) Use prodmgr to uninstall a version of SAT.
This command will do three things:
+cray-product-catalog Kubernetes ConfigMap, so that it will no longer show up
+in the output of sat showrev.prodmgr uninstall sat 2.2.10
+Example output:
+Repository sat-2.2.10-sle-15sp2 has been removed.
+Removed Docker image cray/cray-sat:3.9.0
+Removed Docker image cray/sat-cfs-install:1.0.2
+Removed Docker image cray/sat-install-utility:1.4.0
+Deleted sat-2.2.10 from product catalog.
+This procedure can be used to downgrade the active version of SAT.
+Note: The prodmgr activate command is deprecated in SAT 2.6, and the
+ability to switch between SAT versions will be removed in a future release.
prodmgr command is
+available.(ncn-m001#) Use sat showrev to list versions of SAT.
sat showrev --products --filter product_name=sat
+Example output:
+###############################################################################
+Product Revision Information
+###############################################################################
++--------------+-----------------+--------------------+-----------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+--------------------+-----------------------+
+| sat | 2.3.3 | - | - |
+| sat | 2.2.10 | - | - |
++--------------+-----------------+--------------------+-----------------------+
+(ncn-m001#) Use prodmgr to switch to a different version of SAT.
This command will do two things:
+2.2.10
+sets the repository sat-2.2.10-sle-15sp2 as the only member of the sat-sle-15sp2 group.management-23.5.0). Specifically, it will ensure that the layer refers to the version of SAT CFS
+configuration content associated with the version of SAT to which the system is switching.prodmgr activate sat 2.5.15
+Example output:
+Repository sat-2.5.15-sle-15sp4 is now the default in sat-sle-15sp4.
+Updated CFS configurations: [management-23.5.0]
+Apply the modified CFS configuration to the management NCNs.
+At this point, Nexus package repositories have been modified to set a +particular package repository as active, but the SAT package may not have +been updated on management NCNs.
+To ensure that management NCNs have been updated to use the active SAT +version, follow the Procedure to Apply CFS Configuration.
+(ncn-m001#) Set an environment variable that refers to the name of the CFS configuration
+to be applied to the management NCNs.
export CFS_CONFIG_NAME="management-23.5.0"
+Note: Refer to the output from the prodmgr activate command to find
+the name of the modified CFS configuration. If more than one CFS configuration
+was modified, use the first one.
INFO: Successfully saved CFS configuration "management-23.5.0"
+(ncn-m001#) Obtain the name of the CFS configuration layer for SAT and save it in an
+environment variable:
export SAT_LAYER_NAME=$(cray cfs configurations describe $CFS_CONFIG_NAME --format json \
+ | jq -r '.layers | map(select(.cloneUrl | contains("sat-config-management.git")))[0].name')
+(ncn-m001#) Create a CFS session that executes only the SAT layer of the given CFS
+configuration.
The --configuration-limit option limits the configuration session to run
+only the SAT layer of the configuration.
cray cfs sessions create --name "sat-session-${CFS_CONFIG_NAME}" --configuration-name \
+ "${CFS_CONFIG_NAME}" --configuration-limit "${SAT_LAYER_NAME}"
+Monitor the progress of the CFS session.
+(ncn-m001#) Set an environment variable to name of the Ansible container within the pod
+for the CFS session:
export ANSIBLE_CONTAINER=$(kubectl get pod -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME} -o json \
+ -o json | jq -r '.items[0].spec.containers | map(select(.name | contains("ansible"))) | .[0].name')
+(ncn-m001#) Next, get the logs for the Ansible container.
kubectl logs -c $ANSIBLE_CONTAINER --tail 100 -f -n services \
+ --selector=cfsession=sat-session-${CFS_CONFIG_NAME}
+Ansible plays, which are run by the CFS session, will install SAT on all the +master management NCNs on the system. A summary of results can be found at +the end of the log output.
+(ncn-m001#) The following example shows a successful session:
...
+PLAY RECAP *********************************************************************
+x3000c0s1b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s3b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+x3000c0s5b0n0 : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
+Note: Ensure that the PLAY RECAPs for each session show successes for all +manager NCNs before proceeding.
+(ncn-m001#) Verify that SAT was successfully configured.
If sat is configured, the --version command will indicate which version
+is installed. If sat is not properly configured, the command will fail.
Note: This version number will differ from the version number of the SAT
+release distribution. This is the semantic version of the sat Python package,
+which is different from the version number of the overall SAT release distribution.
sat --version
+Example output:
+sat 3.7.0
+Note: Upon first running sat, there might be additional output while
+the sat container image is downloaded. This occurs the first time sat
+is run on each manager NCN. For example, when running sat for the first time
+on ncn-m001 and then for the first time on ncn-m002, this additional
+output is seen both times.
Trying to pull registry.local/cray/cray-sat:3.7.0-20210514024359_9fed037...
+Getting image source signatures
+Copying blob da64e8df3afc done
+Copying blob 0f36fd81d583 done
+Copying blob 12527cf455ba done
+...
+sat 3.7.0
+(ncn-m001#) Stop the typescript.
exit
+SAT version x.y.z is now installed and configured:
The previous procedure is not always necessary because the CFS Batcher service +automatically detects configuration changes and will automatically create new +sessions to apply configuration changes according to certain rules. For more +information on these rules, refer to Configuration Management with +the CFS Batcher in the Cray System Management Documentation.
+The main scenario in which the CFS batcher will not automatically re-apply the
+SAT layer is when the commit hash of the sat-config-management git repository
+has not changed between SAT versions. The previous procedure ensures the
+configuration is re-applied in all cases, and it is harmless if the batcher has
+already applied an updated configuration.
The Install and Upgrade Framework (IUF) provides commands which install, +upgrade, and deploy products on systems managed by CSM. IUF capabilities are +described in detail in the IUF +section of the +Cray System Management Documentation. +The initial install and upgrade workflows described in the +HPE Cray EX System Software Stack Installation and Upgrade Guide for CSM +(S-8052) detail when and how to use +IUF with a new release of SAT or any other HPE Cray EX product.
+This document does not replicate install, upgrade, or deployment procedures +detailed in the Cray System Management +Documentation. This document provides +details regarding software and configuration content specific to SAT which is +needed when installing, upgrading, or deploying a SAT release. The Cray +System Management Documentation will +indicate when sections of this document should be referred to for detailed +information.
+IUF will perform the following tasks for a release of SAT.
+deliver-product stage:
+update-vcs-config stage:
+update-cfs-config stage:
+prepare-images stage:
+management-nodes-rollout stage:
+IUF uses a variety of CSM and SAT tools when performing these tasks. The IUF +section of the +Cray System Management Documentation +describes how to use these tools directly if it is desirable to use them +instead of IUF.
+This section describes SAT details that an administrator must be aware of +before running IUF stages. Entries are prefixed with Information if no +administrative action is required or Action if an administrator needs +to perform tasks outside of IUF.
+Information: This stage is only run if a VCS working branch is specified for +SAT. By default, SAT does not create or specify a VCS working branch.
+Information: This stage only applies to the management configuration and +not to the managed configuration.
+Information: This stage only applies to management images and not to +managed images.
+After upgrading SAT with IUF, it is recommended to complete the following +procedures before using SAT:
+ +...) in shell output indicate omitted lines.x.y.z with the version of the SAT product stream
+being upgraded.After upgrading SAT, if using the configuration file from a previous version, there may be
+configuration file sections no longer used in the new version. For example, when upgrading
+from Shasta 1.4 to Shasta 1.5, the [redfish] configuration file section is no longer used.
(ncn-m001#) In that case, the following warning may appear upon running sat commands.
WARNING: Ignoring unknown section 'redfish' in config file.
+Remove the [redfish] section from /root/.config/sat/sat.toml to resolve the warning.
[redfish]
+username = "admin"
+password = "adminpass"
+Repeat this process for any configuration file sections for which there are “unknown section” warnings.
+As of SAT version 2.2, some command output that was previously printed to stdout
+is now logged to stderr. These messages are logged at the INFO level. The
+default logging threshold was changed from WARNING to INFO to accommodate
+this logging change. Additionally, some messages previously logged at the INFO
+are now logged at the DEBUG level.
These changes take effect automatically. However, if the default output threshold
+has been manually set in ~/.config/sat/sat.toml, it should be changed to ensure
+that important output is shown in the terminal.
(ncn-m001#) In the following example, the stderr log level, logging.stderr_level, is set to
+WARNING, which will exclude INFO-level logging from terminal output.
grep -A 3 logging ~/.config/sat/sat.toml
+Example output:
+[logging]
+...
+stderr_level = "WARNING"
+To enable the new default behavior, comment this line out, delete it, or set +the value to “INFO”.
+If logging.stderr_level is commented out, its value will not affect logging
+behavior. However, it may be helpful to set its value to INFO as a reminder of
+the new default behavior.
The following commands trigger messages that have been changed from stdout
+print calls to INFO-level (or WARNING- or ERROR-level) log messages:
sat bootsys --stage shutdown --stage session-checkssat sensorsThe following commands trigger messages that have been changed from INFO-level
+log messages to DEBUG-level log messages:
sat nid2xnamesat xname2nidsat swapHPE service representatives use system revision information data to identify +systems in support cases.
+This procedure is not required if SAT was upgraded from 2.1 (Shasta v1.5) +or later. It is required if SAT was upgraded from 2.0 (Shasta v1.4) or +earlier.
+Set System Revision Information.
+(ncn-m001#) Run sat setrev and follow the prompts to set the following site-specific values:
Tip: For “System type”, a system with any liquid-cooled components should be +considered a liquid-cooled system. In other words, “System type” is EX-1C.
+sat setrev
+Example output:
+--------------------------------------------------------------------------------
+Setting: Serial number
+Purpose: System identification. This will affect how snapshots are
+ identified in the HPE backend services.
+Description: This is the top-level serial number which uniquely identifies
+ the system. It can be requested from an HPE representative.
+Valid values: Alpha-numeric string, 4 - 20 characters.
+Type: <class 'str'>
+Default: None
+Current value: None
+--------------------------------------------------------------------------------
+Please do one of the following to set the value of the above setting:
+ - Input a new value
+ - Press CTRL-C to exit
+...
+Verify System Revision Information.
+(ncn-m001#) Run sat showrev and verify the output shown in the “System Revision Information table.”
The following example shows sample table output.
+sat showrev
+Example output:
+################################################################################
+System Revision Information
+################################################################################
++---------------------+---------------+
+| component | data |
++---------------------+---------------+
+| Company name | HPE |
+| Country code | US |
+| Interconnect | Sling |
+| Product number | R4K98A |
+| Serial number | 12345 |
+| Site name | HPE |
+| Slurm version | slurm 20.02.5 |
+| System description | Test System |
+| System install date | 2021-01-29 |
+| System name | eniac |
+| System type | EX-1C |
++---------------------+---------------+
+################################################################################
+Product Revision Information
+################################################################################
++--------------+-----------------+------------------------------+------------------------------+
+| product_name | product_version | images | image_recipes |
++--------------+-----------------+------------------------------+------------------------------+
+| csm | 0.8.14 | cray-shasta-csm-sles15sp1... | cray-shasta-csm-sles15sp1... |
+| sat | 2.0.1 | - | - |
+| sdu | 1.0.8 | - | - |
+| slingshot | 0.8.0 | - | - |
+| sma | 1.4.12 | - | - |
++--------------+-----------------+------------------------------+------------------------------+
+################################################################################
+Local Host Operating System
+################################################################################
++-----------+----------------------+
+| component | version |
++-----------+----------------------+
+| Kernel | 5.3.18-24.15-default |
+| SLES | SLES 15-SP2 |
++-----------+----------------------+
+By default, SAT uses Boot Orchestration Service (BOS) version two (v2).
+Select the BOS version to use for individual commands with the --bos-version
+option. For more information on this option, refer to the man page for a specific
+command.
Another way to change the BOS version is by configuring it under the
+api_version setting in the bos section of the SAT configuration file.
+If the system is using an existing SAT configuration file from an older
+version of SAT, the bos section might not exist. In that case, add the bos
+section with the BOS version desired in the api_version setting.
Find the SAT configuration file at ~/.config/sat/sat.toml, and look for a
+section like this:
[bos]
+api_version = "v2"
+In this example, SAT is using BOS version "v2".
Change the line specifying the api_version to the BOS version desired (for
+example, "v1").
[bos]
+api_version = "v1"
+If applicable, uncomment the api_version line.
If the system is using an existing SAT configuration file from a recent
+version of SAT, the api_version line might be commented out like this:
[bos]
+# api_version = "v2"
+If the line is commented out, SAT will still use the default BOS
+version. To ensure a different BOS version is used, uncomment the
+api_version line by removing # at the beginning of the line.
SAT supports supplying tenant information to CSM services in order to allow +tenant admins to use SAT within their tenant. By default, the tenant name is +not set, and SAT will not send any tenant information with its requests to +CSM services. Configure the tenant name either in the SAT configuration file +or on the command line.
+Set the tenant name in the SAT configuration file using the
+api_gateway.tenant_name option.
Here is an example:
+[api_gateway]
+tenant_name = "my_tenant"
+Set the tenant name for each sat invocation using the --tenant-name
+option. The --tenant-name option must be specified before the subcommand
+name.
(ncn-m001#) Here is an example:
sat --tenant-name=my_tenant status
+The Install and Upgrade Framework (IUF) provides commands which install,
+upgrade, and deploy products on systems managed by CSM with the help of
+sat bootprep. Outside of IUF, it is uncommon to use sat bootprep.
+For more information on IUF, see the
+IUF section of
+the Cray System Management Documentation.
+For more information on sat bootprep, see SAT Bootprep.
Both IUF and sat bootprep allow variable substitutions into the default HPC
+CSM Software Recipe bootprep input files. The default variables of the HPC
+CSM Software Recipe are available in a product_vars.yaml file. To override
+the default variables, specify any site variables in a site_vars.yaml file.
+Variables are sourced from the command line, any variable files directly
+provided, and the HPC CSM Software Recipe files used, in that order.
IUF also has special session variables internal to the iuf command that
+override any matching entries. Session variables are the set of product and
+version combinations being installed by the current IUF activity, and they are
+found inside IUF’s internal session_vars.yaml file. For more information on
+IUF and variable substitutions, see the
+IUF section of
+the Cray System Management Documentation.
When using sat bootprep outside of IUF, substituting variables into the
+default bootprep input files might cause problems. Complex variables like
+"{{ working_branch }}" cannot be completely resolved outside of IUF and
+its internal session variables. Thus, the default product_vars.yaml file is
+unusable with only the sat bootprep command when variables like
+"{{ working_branch }}" are used. To work around this limitation when
+substituting complex variables, use the internal IUF session_vars.yaml file
+with sat bootprep and the default bootprep input files.
Find the session_vars.yaml file from the most recent IUF activity on the
+system.
This process is documented in the upgrade prerequisites procedure of the +Cray System Management Documentation. For more information, see steps 1-6 of +Stage 0.3 - Option 2.
+(ncn-m001#) Use the session_vars.yaml file to substitute variables into the default
+bootprep input files.
sat bootprep run --vars-file session_vars.yaml
+The sat bootprep run command uses information from the bootprep input files
+to create CFS configurations, IMS images, and BOS session templates. To restrict
+this creation into separate stages, use the --limit option and list whether
+to create configurations, images, session_templates, or some
+combination of these. IUF uses the --limit option in this way to install,
+upgrade, and deploy products on a system in stages.
(ncn-m001#) For example, to create only CFS configurations, run the following command used
+by the IUF update-cfs-config stage:
sat bootprep run --limit configurations example-bootprep-input-file.yaml
+Example output:
+INFO: Validating given input file example-bootprep-input-file.yaml
+INFO: Input file successfully validated against schema
+INFO: Creating 3 CFS configurations
+...
+INFO: Skipping creation of IMS images based on value of --limit option.
+INFO: Skipping creation of BOS session templates based on value of --limit option.
+(ncn-m001#) To create only IMS images and BOS session templates, run the following command
+used by the IUF prepare-images stage:
sat bootprep run --limit images --limit session_templates example-bootprep-input-file.yaml
+Example output:
+INFO: Validating given input file example-bootprep-input-file.yaml
+INFO: Input file successfully validated against schema
+INFO: Skipping creation of CFS configurations based on value of --limit option.
+SAT provides an automated solution for creating CFS configurations, building
+and configuring images in IMS, and creating BOS session templates. The
+solution is based on a given input file that defines how those configurations,
+images, and session templates should be created. This automated process centers
+around the sat bootprep command. Man page documentation for sat bootprep
+can be viewed similar to other SAT commands.
(ncn-m001#) Here is an example:
sat-man sat-bootprep
+The sat bootprep command helps the Install and Upgrade Framework (IUF)
+install, upgrade, and deploy products on systems managed by CSM. Outside of IUF,
+it is uncommon to use sat bootprep. For more information on this relationship,
+see SAT and IUF. For more information on IUF, see the
+IUF section of
+the Cray System Management Documentation.
sat bootprep is used to create CFS configurations, build and
+rename IMS images, and create BOS session templates which tie the
+configurations and images together during a BOS session.
sat bootsys automates several portions of the boot and shutdown processes,
+including (but not limited to) performing BOS operations (such as creating BOS
+sessions), powering on and off cabinets, and checking the state of the system
+prior to shutdown.
The input file provided to sat bootprep is a YAML-formatted file containing
+information which CFS, IMS, and BOS use to create configurations, images, and
+BOS session templates respectively. Writing and modifying these input files is
+the main task associated with using sat bootprep. An input file is composed of
+three main sections, one each for configurations, images, and session templates.
+These sections may be specified in any order, and any of the sections may be
+omitted if desired.
The sat bootprep input file is validated against a versioned schema
+definition. The input file should specify the version of the schema with which
+it is compatible under a schema_version key. For example:
---
+schema_version: 1.0.2
+(ncn-m001#) The current sat bootprep input file schema version can be viewed with the
+following command:
sat bootprep view-schema | grep '^version:'
+Example output:
+version: '1.0.2'
+The sat bootprep run command validates the schema version specified
+in the input file. The command also makes sure that the schema version
+of the input file is compatible with the schema version understood by the
+current version of sat bootprep. For more information on schema version
+validation, refer to the schema_version property description in the bootprep
+input file schema. For more information on viewing the bootprep input file
+schema in either raw form or user-friendly HTML form, see View SAT Bootprep
+Schema.
The default HPC CSM Software Recipe bootprep input files provided by the
+hpc-csm-software-recipe release distribution already contain the correct
+schema version.
The CFS configurations are defined under a configurations key. Under this
+key, list one or more configurations to create. For each
+configuration, give a name in addition to the list of layers that
+comprise the configuration.
Each layer can be defined by a product name and optionally a version number,
+commit hash, or branch in the product’s configuration repository. If this
+method is used, the layer is created in CFS by looking up relevant configuration
+information (including the configuration repository and commit information) from
+the cray-product-catalog Kubernetes ConfigMap as necessary. A version may be
+supplied. However, if it is absent, the version is assumed to be the latest
+version found in the cray-product-catalog.
Alternatively, a configuration layer can be defined by explicitly referencing
+the desired configuration repository. Specify the intended version
+of the Ansible playbooks by providing a branch name or commit hash with branch
+or commit.
The following example shows a CFS configuration with two layers. The first +layer is defined in terms of a product name and version, and the second layer +is defined in terms of a Git clone URL and branch:
+---
+configurations:
+- name: example-configuration
+ layers:
+ - name: example-product
+ playbook: example.yml
+ product:
+ name: example
+ version: 1.2.3
+ - name: another-example-product
+ playbook: another-example.yml
+ git:
+ url: "https://vcs.local/vcs/another-example-config-management.git"
+ branch: main
+When sat bootprep is run against an input file, a CFS configuration is created
+corresponding to each configuration in the configurations section. For
+example, the configuration created from an input file with the layers listed
+above might look something like the following:
{
+ "lastUpdated": "2022-02-07T21:47:49Z",
+ "layers": [
+ {
+ "cloneUrl": "https://vcs.local/vcs/example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "example product",
+ "playbook": "example.yml"
+ },
+ {
+ "cloneUrl": "https://vcs.local/vcs/another-example-config-management.git",
+ "commit": "<commit hash>",
+ "name": "another example product",
+ "playbook": "another-example.yml"
+ }
+ ],
+ "name": "example-configuration"
+}
+The IMS images are defined under an images key. Under the images key, the
+user may define one or more images to be created in a list. Each element of the
+list defines a separate IMS image to be built and/or configured. Images must
+contain a name key and a base key.
The name key defines the name of the resulting IMS image. The base key
+defines the base image to be configured or the base recipe to be built and
+optionally configured. One of the following keys must be present under the
+base key:
ims key to specify an existing image or recipe in IMS.product key to specify an image or recipe provided by a particular
+version of a product. If a product provides more than one image or recipe,
+specify a filter to select one. For more information, see
+Filter Base Images or Recipes from a Product.image_ref key to specify another image from the input file
+using its ref_name.Images may also contain the following keys:
+configuration key to specify a CFS configuration with which to
+customize the built image. If a configuration is specified, then configuration
+groups must also be specified using the configuration_group_names key.ref_name key to specify a unique name that can refer to this image
+within the input file in other images or in session templates. The ref_name
+key allows references to images from the input file that have dynamically
+generated names as described in
+Dynamic Variable Substitutions.description key to describe the image in the bootprep input file.
+Note that this key is not currently used.Here is an example of an image using an existing IMS recipe as its base. This
+example builds an IMS image from that recipe. It then configures it with
+a CFS configuration named example-compute-config. The example-compute-config
+CFS configuration can be defined under the configurations key in the same
+input file, or it can be an existing CFS configuration. Running sat bootprep
+against this input file results in an image named example-compute-image.
images:
+- name: example-compute-image
+ description: >
+ An example compute node image built from an existing IMS recipe.
+ base:
+ ims:
+ name: example-compute-image-recipe
+ type: recipe
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+Here is an example showing the definition of two images. The first image is
+built from a recipe provided by the uss product. The second image uses the
+first image as a base and configures it with a configuration named
+example-compute-config. The value of the first image’s ref_name key is used
+in the second image’s base.image_ref key to specify it as a dependency.
+Running sat bootprep against this input file results in two images, the
+first named example-uss-image and the second named example-compute-image.
images:
+- name: example-uss-image
+ ref_name: example-uss-image
+ description: >
+ An example image built from the recipe provided by the USS product.
+ base:
+ product:
+ name: uss
+ version: 1.0.0
+ type: recipe
+- name: example-compute-image
+ description: >
+ An example image that is configured from an image built from the recipe provided
+ by the USS product.
+ base:
+ image_ref: example-uss-image
+ configuration: example-compute-config
+ configuration_group_names:
+ - Compute
+This example assumes that the given version of the uss product provides
+only a single IMS recipe. If more than one recipe is provided by the
+given version of the uss product, use a filter as described in
+Filter Base Images or Recipes from a Product.
A product may provide more than one image or recipe. If this happens,
+filter the product’s images or recipes whenever a base image or recipe from
+that product is used. Beneath the base.product value within an image,
+specify a filter key to create a filter using the following criteria:
prefix key to filter based on a prefix matching the name of the
+image or recipe.wildcard key to filter based on a shell-style wildcard matching the
+name of the image or recipe.arch key to filter based on the target architecture of the image or
+recipe in IMS.When specifying more than one filter key, all filters must match only the +desired image or recipe. An error occurs if either no images or recipes +match the given filters or if more than one image or recipe matches +the given filters.
+Here is an example of three IMS images built from the Kubernetes image and the
+Ceph storage image provided by the csm product. This example uses a prefix
+filter to select from the multiple images provided by the CSM product.
+The first two IMS images in the example find any image from the specified csm
+product version whose name starts with secure-kubernetes. The third image in
+the example finds any csm image whose name starts with secure-storage-ceph.
+All three images are then configured with a configuration named
+example-management-config. Running sat bootprep against this input file
+results in three IMS images named worker-example-csm-image,
+master-example-csm-image, and storage-example-csm-image.
images:
+- name: worker-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-kubernetes
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Worker
+
+- name: master-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-kubernetes
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Master
+
+- name: storage-example-csm-image
+ base:
+ product:
+ name: csm
+ version: 1.4.1
+ type: image
+ filter:
+ prefix: secure-storage-ceph
+ configuration: example-management-config
+ configuration_group_names:
+ - Management_Storage
+Here is an example of two IMS images built from recipes provided by the uss
+product. This example uses an architecture filter to select from the multiple
+recipes provided by the USS product. The first image will be built from the
+x86_64 version of the IMS recipe provided by the specified version of the
+uss product. The second image will be built from the aarch64 version of
+the IMS recipe provided by the specified version of the uss product.
images:
+- name: example-uss-image-x86_64
+ ref_name: example-uss-image-x86_64
+ description: >
+ An example image built from the x86_64 recipe provided by the USS product.
+ base:
+ product:
+ name: uss
+ version: 1.0.0
+ type: recipe
+ filter:
+ arch: x86_64
+
+- name: example-uss-image-aarch64
+ ref_name: example-uss-image-aarch64
+ description: >
+ An example image built from the aarch64 recipe provided by the USS product.
+ base:
+ product:
+ name: uss
+ version: 1.0.0
+ type: recipe
+ filter:
+ arch: aarch64
+The BOS session templates are defined under the session_templates key. Each
+session template must provide values for the name, image, configuration,
+and bos_parameters keys. The name key defines the name of the resulting BOS
+session template. The image key defines the image to use in the BOS session
+template. One of the following keys must be present under the image key:
ims key to specify an existing image or recipe in IMS.image_ref key to specify another image from the input file
+using its ref_name.The configuration key defines the CFS configuration specified
+in the BOS session template.
The bos_parameters key defines parameters that are passed through directly to
+the BOS session template. The bos_parameters key should contain a boot_sets
+key, and each boot set in the session template should be specified under
+boot_sets. Each boot set can contain the following keys, all of
+which are optional:
arch key to specify the architecture of the nodes that should be
+targeted by the boot set. Valid values are the same as those used by
+Hardware State Manager (HSM).kernel_parameters key to specify the parameters passed to the kernel
+on the command line.network key to specify the network over which the nodes boot.node_list key to specify the nodes to add to the boot set.node_roles_groups key to specify the HSM roles to add to the boot
+set.node_groups key to specify the HSM groups to add to the boot set.rootfs_provider key to specify the root file system provider.rootfs_provider_passthrough key to specify the parameters to add to
+the rootfs= kernel parameter.As mentioned above, the parameters under bos_parameters are passed through
+directly to BOS. For more information on the properties of a BOS boot set,
+refer to BOS Session Templates in the Cray
+System Management Documentation.
Here is an example of a BOS session template that refers to an existing IMS
+image by name and targets nodes with the role Compute and the architecture
+X86 in HSM:
session_templates:
+- name: example-session-template
+ image:
+ ims:
+ name: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ arch: X86
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+Here is an example of a BOS session template that refers to an image from the
+input file by its ref_name and targets nodes with the role Compute and the
+architecture ARM in HSM. Note that using the image_ref key requires that
+an image defined in the input file specifies example-image as the value of
+its ref_name key.
session_templates:
+- name: example-session-template
+ image:
+ image_ref: example-image
+ configuration: example-configuration
+ bos_parameters:
+ boot_sets:
+ example_boot_set:
+ arch: ARM
+ kernel_parameters: ip=dhcp quiet
+ node_roles_groups:
+ - Compute
+ rootfs_provider: cpss3
+ rootfs_provider_passthrough: dvs:api-gw-service-nmn.local:300:nmn0
+The sat bootprep command takes any variables provided and substitutes them
+into the input file. Variables are sourced from the command line, any variable
+files directly provided, and the HPC CSM Software Recipe files used, in that
+order. When providing values through a variable file, sat bootprep
+substitutes the values with Jinja2 template syntax. The HPC CSM Software Recipe
+provides default variables in a product_vars.yaml variable file. This file
+defines information about each HPC software product included in the recipe.
Variables are primarily substituted into the default HPC CSM Software Recipe
+bootprep input files through IUF. However, variable files can also be given to
+sat bootprep directly from IUF’s use of the recipe. When using variables
+directly with sat bootprep, there are some limitations. For more
+information on SAT variable limitations, see SAT and IUF.
+For more information on IUF and variable substitutions, see the
+IUF section of
+the Cray System Management Documentation.
View a listing of the default HPC CSM Software Recipe variables and
+their values by running sat bootprep list-vars. For more information on
+options that can be used with the list-vars subcommand, refer to the man page
+for the sat bootprep subcommand.
By default, the sat bootprep command uses the variables from the latest
+installed version of the HPC CSM Software Recipe. Override this with the
+--recipe-version command line argument to sat bootprep run.
(ncn-m001#) For example, to explicitly select the 22.11.0 version of the HPC CSM Software
+Recipe default variables, specify --recipe-version 22.11.0:
sat bootprep run --recipe-version 22.11.0 compute-and-uan-bootprep.yaml
+The entire sat bootprep input file is not rendered by the Jinja2 template
+engine. Jinja2 template rendering of the input file is performed individually
+for each supported value. The values of the following keys in the bootprep
+input file support rendering as a Jinja2 template and thus support variables:
name key of each configuration under the configurations key.layers key in a
+configuration:
+nameplaybookgit.branchproduct.versionproduct.branchimages key:
+namebase.product.versionbase.product.filter.archbase.product.filter.prefixbase.product.filter.wildcardconfigurationsession_templates key:
+nameconfigurationYou can use Jinja2 built-in filters in values of any of the keys listed above. +In addition, Python string methods can be called on the string variables.
+Variable names with hyphens are not allowed in Jinja2 expressions because they
+are parsed as an arithmetic expression instead of a single variable. To support
+product names with hyphens, sat bootprep converts hyphens to underscores in
+all top-level keys of the default HPC CSM Software Recipe variables. It also
+converts any variables sourced from the command line or any variable files
+provided directly. When referring to a variable with hyphens in the bootprep
+input file, keep this in mind. For example, to refer to the product version
+variable for slingshot-host-software in the bootprep input file, write
+"{{slingshot_host_software.version}}".
The following example bootprep input file shows how a variable of a USS version +can be used in an input file that creates a CFS configuration for computes. +Only one layer is shown for brevity.
+---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: uss-compute-{{uss.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: uss
+ version: "{{uss.version}}"
+ branch: "{{uss.working_branch}}"
+Note: When the value of a key in the bootprep input file is a Jinja2 +expression, it must be quoted to pass YAML syntax checking.
+Jinja2 expressions can also use filters and Python’s built-in string methods to
+manipulate the variable values. For example, suppose only the major and minor
+components of a USS version are to be used in the branch name for the USS
+layer of the CFS configuration. Use the split string method to
+achieve this as follows:
---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: uss-compute-{{uss.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: uss
+ version: "{{uss.version}}"
+ branch: integration-{{uss.version.split('.')[0]}}-{{uss.version.split('.')[1]}}
+Additional variables are available besides the default variables provided by +the HPC CSM Software Recipe. (For more information, see HPC CSM Software +Recipe Variable Substitutions.) +These additional variables are dynamic because their values are determined +at run-time based on the context in which they appear. Available dynamic +variables include the following:
+The variable base.name can be used in the name of an image under the
+images key. The value of this variable is the name of the IMS image or
+recipe used as the base of this image.
The variable image.name can be used in the name of a session template
+under the session_templates key. The value of this variable is the name of
+the IMS image used in this session template.
Note: The name of a session template is restricted to 45 characters. Keep
+this in mind when using image.name in the name of a session template.
These variables reduce the need to duplicate values throughout the sat bootprep input file and make the following use cases possible:
This section provides an example bootprep input file. It also gives +instructions for obtaining the default bootprep input files delivered +with a release of the HPC CSM Software Recipe.
+The following bootprep input file provides an example of using most of the +features described in previous sections. It is not intended to be a complete +bootprep file for the entire CSM product.
+---
+configurations:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ layers:
+ - name: uss-compute-{{uss.working_branch}}
+ playbook: cos-compute.yml
+ product:
+ name: uss
+ version: "{{uss.version}}"
+ branch: "{{uss.working_branch}}"
+ - name: cpe-pe_deploy-{{cpe.working_branch}}
+ playbook: pe_deploy.yml
+ product:
+ name: cpe
+ version: "{{cpe.version}}"
+ branch: "{{cpe.working_branch}}"
+
+images:
+- name: "{{default.note}}{{base.name}}{{default.suffix}}"
+ ref_name: base_uss_image
+ base:
+ product:
+ name: uss
+ type: recipe
+ version: "{{uss.version}}"
+
+- name: "compute-{{base.name}}"
+ ref_name: compute_image
+ base:
+ image_ref: base_uss_image
+ configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ configuration_group_names:
+ - Compute
+
+session_templates:
+- name: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ image:
+ image_ref: compute_image
+ configuration: "{{default.note}}compute-{{recipe.version}}{{default.suffix}}"
+ bos_parameters:
+ boot_sets:
+ compute:
+ kernel_parameters: ip=dhcp quiet spire_join_token=${SPIRE_JOIN_TOKEN}
+ node_roles_groups:
+ - Compute
+ rootfs_provider_passthrough: "dvs:api-gw-service-nmn.local:300:hsn0,nmn0:0"
+Default bootprep input files are delivered by the HPC CSM Software Recipe
+product. Access these files by cloning the hpc-csm-software-recipe
+repository, as described in the Accessing sat bootprep files process of
+the Cray System Management
+Documentation.
(ncn-m001#) Find the default input files in the bootprep directory of the
+cloned repository:
ls bootprep/
+The sat bootprep generate-example command was not updated for
+recent bootprep schema changes. It is recommended to instead use the
+default bootprep input files described in Access Default Bootprep Input
+Files. The sat bootprep generate-example command will be updated in a future release of SAT.
The sat bootprep run command uses information from the bootprep input file to
+create CFS configurations, IMS images, and BOS session templates. For easy
+reference, the command also includes output summarizing the final creation
+results.
(ncn-m001#) Here is a sample table output after running sat bootprep run:
################################################################################
+CFS configurations
+################################################################################
++------------------+
+| name |
++------------------+
+| example-config-1 |
+| example-config-2 |
++------------------+
+################################################################################
+IMS images
+################################################################################
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+| name | preconfigured_image_id | final_image_id | configuration | configuration_group_names |
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+| example-image | c1bcaf00-109d-470f-b665-e7b37dedb62f | a22fb912-22be-449b-a51b-081af2d7aff6 | example-config | Compute |
++---------------+--------------------------------------+--------------------------------------+----------------+----------------------------+
+################################################################################
+BOS session templates
+################################################################################
++------------------+----------------+
+| name | configuration |
++------------------+----------------+
+| example-template | example-config |
++------------------+----------------+
+The contents of the YAML input files used by sat bootprep must conform to a
+schema which defines the structure of the data. The schema definition is written
+using the JSON Schema format. (Although the format is named “JSON Schema”, the
+schema itself is written in YAML as well.) More information, including introductory
+materials and a formal specification of the JSON Schema metaschema, can be found
+on the JSON Schema website.
(ncn-m001#) To view the exact schema specification, run sat bootprep view-schema.
sat bootprep view-schema
+---
+$schema: "https://json-schema.org/draft/2020-12/schema"
+Example output:
+title: Bootprep Input File
+description: >
+ A description of the set of CFS configurations to create, the set of IMS
+ images to create and optionally customize with the defined CFS configurations,
+ and the set of BOS session templates to create that reference the defined
+ images and configurations.
+type: object
+additionalProperties: false
+properties:
+ ...
+The raw schema definition can be difficult to understand without experience
+working with JSON Schema specifications. For this reason, a feature is included
+with sat bootprep that generates user-friendly HTML documentation for the input
+file schema. This HTML documentation can be browsed with a web browser.
(ncn-m001#) Create a documentation tarball using sat bootprep.
sat bootprep generate-docs
+Example output:
+INFO: Wrote input schema documentation to /root/bootprep-schema-docs.tar.gz
+An alternate output directory can be specified with the --output-dir
+option. The generated tarball is always named bootprep-schema-docs.tar.gz.
sat bootprep generate-docs --output-dir /tmp
+Example output:
+INFO: Wrote input schema documentation to /tmp/bootprep-schema-docs.tar.gz
+(user@hostname>) From another machine, copy the tarball to a local directory.
scp root@ncn-m001:bootprep-schema-docs.tar.gz .
+(user@hostname>) Extract the contents of the tarball and open the contained index.html.
tar xzvf bootprep-schema-docs.tar.gz
+Example output:
+x bootprep-schema-docs/
+x bootprep-schema-docs/index.html
+x bootprep-schema-docs/schema_doc.css
+x bootprep-schema-docs/schema_doc.min.js
+another-machine$ open bootprep-schema-docs/index.html
+