[WIP] update documentation needed for 8.5 release (#197)

update documentation and version numbers.

CHANGELOG.md

@@ -1,6 +1,30 @@
 Open XDMoD SUPReMM Change Log
 =============================
 
+## 2019-10-21 v8.5.0
+- Features
+    - Add Job Efficiency reporting capability. This includes classification
+      of jobs based on performance metrics and components for the new Dashboard
+      tab that show the efficiency metrics by user.
+    - Add ability to export Job Performance data via the new Data Export tab.
+    - Added extra statistics for mounted filesystems `/home`, `/projects` and
+      `/util`. The default source data for these statistics is from the
+      nfs mounted filesystems.
+    - Added ability to configure which devices are used for the various I/O
+      metrics.
+
+- Miscellaneous
+    - Various updates to the module required to support XDMoD 8.5.
+    - The automated CI testing now confirms that the software works with a
+      password protected MongoDB database.
+    - Add more automated CI tests.
+
+- Bug Fixes
+    - Jobs listed in the advance search results in the job viewer and in the show raw data
+      dialog in the metric explorer are now guaranteed to be ordered based on the
+      job end time.
+
+
 ## 2019-04-23 v8.1.0
 - Features
     - Add support for GPU metrics. If available, the GPU usage and GPU memory

configuration/portal_settings.d/supremm.ini

@@ -1,7 +1,7 @@
 [supremm-general]
 
 ; The version number is updated during the upgrade process.
-version = "8.1.0"
+version = "8.5.0"
 
 ; The path of the ETL schema file relative to [share]/etl/js/config/supremm.
 schema_file = "etl.schema.js"

docs/_config.yml

@@ -34,8 +34,8 @@ defaults:
             type: "pages"
         values:
             layout: "page"
-            sw_version: "8.1.0"
-            summ_sw_version: "1.2.0"
+            sw_version: "8.5.0"
+            summ_sw_version: "1.3.0"
             style: "effervescence"
             tocversion: "toc"
 

docs/supremm-install-overview.md

@@ -36,7 +36,7 @@ identical to the version of the base XDMoD install. The versions of the
 summarization software and PCP software for a given XDMoD version are listed
 below.
 
-### Open XDMoD 8.1.0
+### Open XDMoD {{ page.sw_version }}
 
 <table>
 <thead>
@@ -46,10 +46,10 @@ below.
 </thead>
 <tbody>
 <tr>
-<td> Job Summarization </td><td align="right"> {{ page.summ_sw_version }} </td><td align="right"> 1.1.x </td>
+<td> Job Summarization </td><td align="right"> {{ page.summ_sw_version }} </td><td align="right"> 1.1.x, 1.2.x </td>
 </tr>
 <tr>
-<td> PCP               </td><td align="right"> 3.12.2              </td><td align="right"> 3.11.x - 3.12.x </td>
+<td> PCP               </td><td align="right"> 4.1.0 or 4.3.2             </td><td align="right"> 4.x </td>
 </tr>
 </tbody>
 </table>
@@ -58,7 +58,8 @@ below.
 The SUPReMM software has been tested with MongoDB version 3.4.15. We expect
 that the software is compatible with any supported release version of MongoDB.
 
-The summarization software should be compatible with the 4.x releases of PCP.
+The summarization software is tested against the PCP versions shipped with Centos
+7.6 and Centos 7.7. The software should be compatible with the 5.x releases of PCP.
 However the XDMoD team have not tested this configuration and will not be able
 to provide support for it.
 

docs/supremm-processing-upgrade.md

@@ -11,17 +11,20 @@ All periodic cron scripts should be disabled before upgrading the software.
 
 ## Upgrade Software
 
-The upgrade procedure involves installing the new software package and then running
-an upgrade script.
+The upgrade procedure involves installing the new software package.
 
 ### RPM upgrade
 
-    # yum install supremm-{{ page.summ_sw_version }}-1.x86_64.rpm
+RPMs are provided for Centos 7.6 and Centos 7.7 and are compiled against
+the version of PCP that ships with each distribution (PCP version 4.1.0
+and 4.3.2 respectively).
+
+    # yum install supremm-{{ page.summ_sw_version }}-1.el7_?.x86_64.rpm
 
 ### Source code upgrade
 
     $ source activate supremm
-    $ tar xf {{ page.summ_sw_version }}.tar.gz
-    $ cd {{ page.summ_sw_version }}
+    $ tar xf supremm-{{ page.summ_sw_version }}.tar.gz
+    $ cd supremm-{{ page.summ_sw_version }}
     $ python setup.py install
 

docs/supremm-upgrade.md

@@ -7,26 +7,30 @@ General Upgrade Notes
 
 The Job Performance (SUPReMM) XDMoD module should be upgraded at the same time as the main XDMoD
 software. The upgrade procedure is documented on the [XDMoD upgrade
-page](https://open.xdmod.org/upgrade.html).
+page](https://open.xdmod.org/upgrade.html).  The ingestion and aggregation
+script `aggregate_supremm.sh` **must** be run after the XDMoD software has been
+upgraded.
 
 8.1.0 to 8.5.0 Upgrade Notes
 ----------------------------
 
 - This upgrade includes database schema changes.
     - Modifies `modw_supremm` schema.
+    - Modifies the `modw_aggregates.supremmfact_by_` tables.
 
 The `modw_supremm.job` and `modw_supremm.job_error` table have extra columns to store metrics
 about job I/O. These columns are added to the tables by the upgrade script.
 
 See the [Configuration Guide](supremm-configuration.md#supremm_resourcesjson) for information
 about how to define the data mapping for the new I/O metrics.
 
-Changes to the mapping only affect job metrics ingested after the configuration file
+Changes to the mapping only affect the statistics for job data ingested after the configuration file
 is modified. The metrics for jobs that have already been ingested are not automatically
 updated. To update the data for existing jobs it is necessary to reset the job ingest
 status and then run the ingest and aggregation script.
 
-** If you do not update the data mapping then you do not need to perform these steps **
+**If you do not update the data mapping then you do not need to perform the reset step (1)
+but the aggregation script in step (2) must always be run after an upgrade**
 
 Resetting the job ingest status and re-ingesting the data is done as follows:
 
@@ -41,3 +45,13 @@ run for a resource that had approximately 2 million jobs it took approximately
 2) Run the ingest and aggregation script:
 
     $ aggregate_supremm.sh
+
+### Troubleshooting
+
+The `aggregate_supremm.sh` script has to be run after the upgrade script is run.
+If the aggregate script is not run then the error
+```
+SQLSTATE[42S22]: Column not found: 1054 Unknown column 'jf.netdir_util_write' in 'where clause'
+```
+will be displayed in the XDMoD Usage tab and Metric Explorer tabs when trying to plot
+data for the SUPREMM realm.

xdmod-supremm.spec.in

@@ -58,6 +58,8 @@ rm -rf $RPM_BUILD_ROOT
 %config(noreplace) %{_sysconfdir}/xdmod/supremmconfig.json
 
 %changelog
+* Mon Oct 21 2019 XDMoD <[email protected]> 8.5.0-1.0
+- Release 8.5.0.
 * Tue Apr 23 2019 XDMoD <[email protected]> 8.1.0-1.0
 - Release 8.1.0.
 * Thu Nov 8 2018 XDMoD <[email protected]> 8.0.0-1.0