BFD-1676: Introduce parallel performance regression test suite (#1075)

* Initial parallel baseline locust tests

* Remove PACA tests from baseline tests

* Use Locust's on_start() instead of __init__

* Remove cert code taken from PACA tests

* Introduce URL path helper functions in url_path; add get() method in baseline tests to streamline creating requests

* Add a get_ob() class method in baseline tests

* Simplify test code using get() and get_eob() methods

* Attempt to reduce chances of request caching by randomizing input IDs and setting Cache-Control headers

* Support python versions less than 3.9 by using older type hinting syntax

* Replace merge syntax with dictionary unpacking as Python 3.5 supports it

* Use v2 endpoint for all tests

* Uncomment hashed MBI test

* Uncomment contract cursors test

* Properly handle case where params is empty in get() and create_url_path()

* Add baseline tests for v1 endpoints

* Format v2 baseline_tests.py; shuffle MBIs and cursor URLs in addition to EOBs

* Add SLA validations for V1 and V2 baseline tests

* Create new configuration files for running the v1 and v2 baseline tests against c5xlarge instances

* Allow the user to set the path to the config file via the CLI

* Use a dictionary to hold config file data opposed to a class

This allows for values to be read from the config first and then merged with the CLI

* Remove unnecessary values from baseline YAML configurations

* Revert changes to set the configuration path; add load_from_path() to allow loading (but not writing) of configuration values from a file

* Load configuration data from specified path and merge with passed-in CLI data

* Add configPath CLI argument to the README

* Rewrite outdated README information about database CLI arguments

* Add examples of how to run the tests reading from YAML configuration

* Include testFile CLI argument in example as it is not stored

* Add disclaimer about running the contract cursors generation script

* Remove unnecessary time import in url_path

* Rename eob_ids to bene_ids along with all subsequent derivatives

* Rename both baseline_test.py files to regression_suite.py

* Add a TODO comment about pulling SLAs from metrics

* Rename config files according to new regression suite naming

* Describe --configPath's behavior more accurately in the README

* Fix --configPath not being an optional CLI argument

* Remove falsey-key discard logic when reading YAML config

* Add some documentation comments to each of the regression suites

* Fix typo in v2 regression suite -- get_bene_ids() to get_bene_Id()

* Correctly refer to name of regression test suite files in README

ops/ccs-ops-misc/load_test/README.md

@@ -70,24 +70,34 @@ The tests are run with parameters that specify all the test params and test file
 
 A single-line test will look like this (placeholders to replace in brackets):
 
-    python3 runtests.py --homePath="<home_directory_path>" --clientCertPath="<home_directory_path>/bluebutton-backend-test-data-server-client-test-keypair.pem" --databaseHost="<AWS_DB_host>" --databaseUsername="<db_username_from_keybase>" --databasePassword="<db_password_from_keybase>" --testHost="https://test.bfd.cms.gov" --testFile="./v2/eob_test_id_count.py" --testRunTime="1m" --maxClients="100" --clientsPerSecond="5"
+    python3 runtests.py --homePath="<home_directory_path>" --clientCertPath="<home_directory_path>/bluebutton-backend-test-data-server-client-test-keypair.pem" --databaseUri="postgres://<db-username>:<db-password>@<db-host>:<port>/<db-name>" --testHost="https://test.bfd.cms.gov" --testFile="./v2/eob_test_id_count.py" --testRunTime="1m" --maxClients="100" --clientsPerSecond="5"
+
+If you have existing configuration in `./config.yml`, you can also run the tests via:
+
+```
+python3 runtests.py --testFile="<your-test-file>"
+```
+
+Or, if you have some YAML configuration in a different file (note that these values will be saved to the root `./config.yml`, so subsequent runs can omit the `configPath` if you are not changing anything):
+
+```
+python3 runtests.py --configPath="config/<your-config-here>.yml" --testFile="<your-test-file>" <other-cli-args-here>...
+```
 
 Essentially, all the items you would set up in the config file are set in a single line. There are some optional, and some required parameters here:
 
 **--homePath** : (Required) : The path to your home directory on the local box, such as /home/logan.mitchell/
 
 **--clientCertPath** : (Required) : The path to the PEM file we copied/modified earlier. Should be located in your home directory like this: ~/bluebutton-backend-test-data-server-client-test-keypair.pem
 
-**--databaseHost** : (Required) : The database host, used to pull data. Get this from the database endpoint we noted in the earlier step "Getting The Database Instance"
-
-**--databaseUsername** : (Required) : The username for connecting to the database, can be found in Keybase. Make sure to use the correct one for the environment you're connecting to.
-
-**--databasePassword** : (Required) : The password for connecting to the database, can be found in Keybase. Make sure to use the correct one for the environment you're connecting to.
+**--databaseUri** : (Required) : The URI used for connecting to the database server. Needs to include username, password (both can be found in Keybase, make sure to use the correct one for the environment you're connecting to), hostname and port.
 
 **--testHost** : (Required) : The load balancer or single node to run the tests against. The environment used here should match the same environment (test, prod-sbx, prod) as the database, so we pull the appropriate data for the environment tested against. Note that when using a single node, you should specity the Ip AND the port for the application.
 
 **--testFile** : (Required) : The path to the test file we want to run.
 
+**--configPath** : (Optional) : The path to a YAML configuration file that will be read from for the values specified here. The values in this configuration file will be merged with values from the CLI, with the CLI values taking priority. The resulting merged values will be written to the repository's root `config.yml`, so if `--configPath` is specified as a YAML file other than `config.yml` the YAML file at that path will not be modified (only read from). If not provided, defaults to `config.yml` (the root YAML configuration file). 
+
 **--serverPublicKey** : (Optional) : To allow the tests to trust the server responses, you can add the path to the public certificate here. This is not required to run the tests successfully, and may not be needed as a parameter at all. If not provided, defaults to an empty string (does not cause test issues.)
 
 **--testRunTime** : (Optional) : How long the test will run. This uses values such as 1m, 30s, 1h, or combinations of these such as 1m 30s. If not provided, defaults to 1m. **Note**: We automatically adjust the run time so that the test runs for the specified amount of time *after* spawning all clients (see `--maxClients` and `--clientsPerSecond` below). For example, with the defaults of a one-minute test, 100 clients, and a spawn rate of 5 clients per second, then the script will spend twenty seconds ramping up its clients and *then* run for the one minute specified. It is optional whether or not to use the `--resetStats` flag to drop the statistics covering this ramp-up period.
@@ -104,6 +114,12 @@ Essentially, all the items you would set up in the config file are set in a sing
 
 There is one special type of test that requires a data setup script to be run beforehand; this is the coverage contract test. This test runs through every list of pages for a set of contracts, but the urls for hitting all the pages of whatever contracts need to be set up before the test runs. This can be done by calling the data setup script common/write_contract_cursors.py similar to this:
 
+> **Note:** You _must_ generate the contract URLs via the below script if you are running any of the following test suites:
+> * `v1/patient_test_coverageContract.py`
+> * `v1/regression_suite.py`
+> * `v2/patient_test_coverageContract.py`
+> * `v2/regression_suite.py`
+
     python3 ./common/write_contract_cursors.py --contracts="Z9997,Z9998,Z9999" --year="2020" --month="01" --count="5" --version="v2"
 
 The script will write out the results to homePath specified in the config (and expect them there during runtime.)

ops/ccs-ops-misc/load_test/common/config.py

@@ -1,19 +1,20 @@
+import os
 import yaml
 
 '''
 Saves a config file using the input file data.
 '''
 def save(fileData):
-    configFile = open("config.yml", 'w')
-    configFile.write("homePath: \"%s\"\n" % fileData.homePath)
-    configFile.write("clientCertPath: \"%s\"\n" % fileData.clientCertPath)
-    configFile.write("serverPublicKey: \"%s\"\n" % fileData.serverPublicKey)
-    configFile.write("dbUri: \"%s\"\n" % fileData.dbUri)
-    configFile.write("testHost: \"%s\"\n" % fileData.testHost)
-    configFile.write("testRunTime: \"%s\"\n" % fileData.testRunTime)
-    configFile.write("testNumTotalClients: \"%s\"\n" % fileData.testNumTotalClients)
-    configFile.write("testCreatedClientsPerSecond: \"%s\"\n" % fileData.testCreatedClientsPerSecond)
-    configFile.write("resetStatsAfterClientSpawn: \"%s\"" % fileData.resetStatsAfterClientSpawn)
+    configFile = open('config.yml', 'w')
+    configFile.write("homePath: \"%s\"\n" % fileData["homePath"])
+    configFile.write("clientCertPath: \"%s\"\n" % fileData["clientCertPath"])
+    configFile.write("serverPublicKey: \"%s\"\n" % fileData["serverPublicKey"])
+    configFile.write("dbUri: \"%s\"\n" % fileData["dbUri"])
+    configFile.write("testHost: \"%s\"\n" % fileData["testHost"])
+    configFile.write("testRunTime: \"%s\"\n" % fileData["testRunTime"])
+    configFile.write("testNumTotalClients: \"%s\"\n" % fileData["testNumTotalClients"])
+    configFile.write("testCreatedClientsPerSecond: \"%s\"\n" % fileData["testCreatedClientsPerSecond"])
+    configFile.write("resetStatsAfterClientSpawn: \"%s\"" % fileData["resetStatsAfterClientSpawn"])
     configFile.close()
 
 '''
@@ -23,37 +24,47 @@ def save(fileData):
 '''
 def create():
 
-    ## Create a small data object for holding the input data
-    class fileData: pass
+    ## Create a dictionary for holding the input data
+    fileData = {}
     ## Prompt user for 4 config values and write to file
-    fileData.homePath = input("Input full path to the home directory: ")
-    fileData.clientCertPath = input("Input full path to the client cert file (pem): ")
-    fileData.serverPublicKey = input("Input server public key (optional, hit enter to skip): ")
-    fileData.dbUri = input("Input database uri for environment under test: ")
-    fileData.testHost = input("Input desired test host (BFD server ip+port to test against, ex: https://10.235.16.152:7443 or load balancer address ex. https://test.bfd.cms.gov): ")
-    fileData.testRunTime = input("Input desired test run time (eg. 30s, 1m): ")
-    fileData.testNumTotalClients = input("Input total number of clients to create: ")
-    fileData.testCreatedClientsPerSecond = input("Input number of clients to create per second (ramp-up speed): ")
-    fileData.resetStatsAfterClientSpawn = (input("Reset statistics after spawning clients? [y/N]: ").lower == 'y')
+    fileData["homePath"] = input("Input full path to the home directory: ")
+    fileData["clientCertPath"] = input("Input full path to the client cert file (pem): ")
+    fileData["serverPublicKey"] = input("Input server public key (optional, hit enter to skip): ")
+    fileData["dbUri"] = input("Input database uri for environment under test: ")
+    fileData["testHost"] = input("Input desired test host (BFD server ip+port to test against, ex: https://10.235.16.152:7443 or load balancer address ex. https://test.bfd.cms.gov): ")
+    fileData["testRunTime"] = input("Input desired test run time (eg. 30s, 1m): ")
+    fileData["testNumTotalClients"] = input("Input total number of clients to create: ")
+    fileData["testCreatedClientsPerSecond"] = input("Input number of clients to create per second (ramp-up speed): ")
+    fileData["resetStatsAfterClientSpawn"] = (input("Reset statistics after spawning clients? [y/N]: ").lower == 'y')
     save(fileData)
 
     ## Attempt to read the new file
     try:
-        config = yaml.safe_load(open("config.yml"))
+        config = yaml.safe_load(open('config.yml'))
         return config
     except yaml.YAMLError as err:
         print("Unable to parse YAML configuration file; please check/create the file manually from the sample file.")
     except OSError as err:
         print("Could not read the new file; please try again.")
 
 '''
-Loads a config from the config file; if no file exists, will attempt to create one via user prompts.
+Loads a config from the default config file (./config.yml); if no file exists, will attempt 
+to create one via user prompts.
 
 Returns the loaded config, or None if nothing could be loaded or an error occurred.
 '''
 def load():
+    return load_from_path('config.yml')
+
+'''
+Loads a config from the specified config file path; if no file exists, will attempt to 
+create one via user prompts.
+
+Returns the loaded config, or None if nothing could be loaded or an error occurred.
+'''
+def load_from_path(path: str):
     try:
-        return yaml.safe_load(open("config.yml"))
+        return yaml.safe_load(open(path))
     except yaml.YAMLError as err:
         print("Unable to parse YAML configuration file; please ensure the format matches the example file.")
         return

ops/ccs-ops-misc/load_test/common/url_path.py

@@ -0,0 +1,12 @@
+from typing import Dict
+from urllib.parse import urlencode
+
+'''
+Creates a query path from a base path (i.e. /v2/fhir/Coverage) and a dictionary of query parameters
+'''
+def create_url_path(path: str, query_params: Dict[str, str] = {}) -> str:
+  if not query_params:
+    return path
+
+  params = urlencode(query_params)
+  return "{}?{}".format(path, params)

ops/ccs-ops-misc/load_test/common/validation.py

@@ -9,13 +9,22 @@
 SLA_PATIENT = "SLA_PATIENT"
 SLA_EOB_WITH_SINCE = "SLA_EOB_WITH_SINCE"
 SLA_EOB_WITHOUT_SINCE = "SLA_EOB_WITHOUT_SINCE"
+SLA_V1_BASELINE = "SLA_V1_BASELINE"
+SLA_V2_BASELINE = "SLA_V2_BASELINE"
 
 ## Values are for 50%, 95%, 99%, and the failsafe limit in order, in milliseconds
+## TODO: Pull these values from production metrics (i.e. New Relic) 
 slas = {
     SLA_COVERAGE : [10, 100, 250, 500],
     SLA_PATIENT : [1000, 3000, 5000, 8000],
     SLA_EOB_WITH_SINCE : [100, 250, 1000, 3000],
-    SLA_EOB_WITHOUT_SINCE : [500, 1000, 3000, 6000]
+    SLA_EOB_WITHOUT_SINCE : [500, 1000, 3000, 6000],
+    # The following values were selected for 5 concurrent users against
+    # a target c5xlarge AWS instance running bfd-server 
+    SLA_V1_BASELINE : [10, 325, 550, 10000],
+    # The following values were selected for 3 concurrent users against
+    # a target c5xlarge AWS instance running bfd-server
+    SLA_V2_BASELINE : [10, 325, 550, 10000],
 }
 
 '''

ops/ccs-ops-misc/load_test/config/config.regression_v1_c5xlarge.yml

@@ -0,0 +1,7 @@
+# This configuration was created for targeting a c5xlarge instance with the V1 performance regression suite
+# in v1/regression_suite.py. It was determined that 5 concurrent users puts approximately
+# 50% - 60% load on a machine of that size.
+
+testRunTime: "3m"
+testNumTotalClients: "5"
+testCreatedClientsPerSecond: "5"

ops/ccs-ops-misc/load_test/config/config.regression_v2_c5xlarge.yml

@@ -0,0 +1,7 @@
+# This configuration was created for targeting a c5xlarge instance with the V2 performance regression suite
+# in v2/regression_suite.py. It was determined that 3 concurrent users puts approximately
+# 50% - 60% load on a machine of that size.
+
+testRunTime: "3m"
+testNumTotalClients: "3"
+testCreatedClientsPerSecond: "3"

ops/ccs-ops-misc/load_test/runtests.py

@@ -52,25 +52,33 @@ def adjusted_run_time(runTime, maxClients, clientsPerSecond):
 Runs a specified test via the input args.
 '''
 def run_with_params(argv):
+    ## Dictionary that holds the default values of each config value
+    defaultConfigData = {
+        'homePath': '',
+        'clientCertPath': '',
+        'databaseUri': '',
+        'testHost': '',
+        'configPath': 'config.yml',
+        'serverPublicKey': '',
+        'testRunTime': "1m",
+        'testNumTotalClients': "100",
+        'testCreatedClientsPerSecond': "5",
+        'resetStatsAfterClientSpawn': False
+    }
+
+    ## Dictionary to hold data passed in via the CLI that will be stored 
+    ## in the root config.yml file
+    configData = {}
 
     testFile = ''
-
-    ## container to hold config data
-    class configData: pass
-
-    ## Optional Params with defaults
-    configData.serverPublicKey = ''
-    configData.testRunTime = "1m"
-    configData.testNumTotalClients = "100"
-    configData.testCreatedClientsPerSecond = "5"
-    configData.resetStatsAfterClientSpawn = False
     workerThreads = "1"
 
     helpString = ('runtests.py \n--homePath="<path/to/home/directory>" (Required) '
      '\n--clientCertPath="<path/to/client/pem/file>" (Required)'
      '\n--databaseUri="postgres://<username:password>@<database-aws-node>.rds.amazonaws.com:port/<dbname>" (Required)'
      '\n--testHost="https://<nodeIp>:7443 or https://<environment>.bfd.cms.gov" (Required)'
      '\n--testFile="/<v1/v2>/test_to_run.py" (Required)'
+     '\n--configPath="<path to a YAML configuration that will be read for CLI values but _not_ written to>" (Optional, Default: "./config.yml")'
      '\n--serverPublicKey="<server public key>" (Optional, Default: "")'
      '\n--testRunTime="<Test run time, ex. 30s, 1m, 2d 1h>" (Optional, Default 1m)'
      '\n--maxClients="<Max number of clients to create at once, int>" (Optional, Default 100)'
@@ -80,8 +88,8 @@ class configData: pass
 
     try:
         opts, args = getopt.getopt(argv,"h",["homePath=", "clientCertPath=", "databaseUri=",
-        "testHost=", "serverPublicKey=", "testRunTime=", "maxClients=", "clientsPerSecond=",
-        "testFile=","workerThreads=","resetStats"])
+        "testHost=", "serverPublicKey=", "configPath=", "testRunTime=", "maxClients=",
+        "clientsPerSecond=", "testFile=", "workerThreads=", "resetStats"])
     except getopt.GetoptError:
         print(helpString)
         sys.exit(2)
@@ -91,48 +99,60 @@ class configData: pass
             print(helpString)
             sys.exit()
         elif opt == "--homePath":
-            configData.homePath = arg
+            configData["homePath"] = arg
         elif opt == "--clientCertPath":
-            configData.clientCertPath = arg
+            configData["clientCertPath"] = arg
         elif opt == "--databaseUri":
-            configData.dbUri = arg
+            configData["dbUri"] = arg
         elif opt == "--testHost":
-            configData.testHost = arg
+            configData["testHost"] = arg
+        elif opt == "--configPath":
+            configData["configPath"] = arg
         elif opt == "--serverPublicKey":
-            configData.serverPublicKey = arg
+            configData["serverPublicKey"] = arg
         elif opt == "--testRunTime":
-            configData.testRunTime = arg
+            configData["testRunTime"] = arg
         elif opt == "--maxClients":
-            configData.testNumTotalClients = arg
+            configData["testNumTotalClients"] = arg
         elif opt == "--clientsPerSecond":
-            configData.testCreatedClientsPerSecond = arg
+            configData["testCreatedClientsPerSecond"] = arg
         elif opt == "--testFile":
             testFile = arg
         elif opt == "--workerThreads":
             workerThreads = arg
         elif opt == "--resetStats":
-            configData.resetStatsAfterClientSpawn = True
+            configData["resetStatsAfterClientSpawn"] = True
         else:
             print(helpString)
             sys.exit()
 
+    ## Read the specified configuration file
+    yamlConfig = config.load_from_path(configData.get("configPath", defaultConfigData["configPath"]))
+    ## Merge the stored data with data passed in via the CLI, with the
+    ## CLI data taking priority
+    configData = {**yamlConfig, **configData}
+    ## Finally, merge the merged configuration values with the defaults,
+    ## in case any optional arguments were not set via the CLI or the specified
+    ## YAML configuration file
+    configData = {**defaultConfigData, **configData}
+
     ## Add on extra time to the run-time to account for ramp-up of clients.
-    adjusted_time = adjusted_run_time(configData.testRunTime,
-        configData.testNumTotalClients, configData.testCreatedClientsPerSecond)
+    adjusted_time = adjusted_run_time(configData["testRunTime"],
+        configData["testNumTotalClients"], configData["testCreatedClientsPerSecond"])
     if adjusted_time is None:
         print("Could not determine adjusted run time. Please use a format " +
             "like \"1m 30s\" for the --testRunTime option")
         sys.exit(1)
-    configData.testRunTime = f"{adjusted_time}s"
+    configData["testRunTime"] = f"{adjusted_time}s"
     print('Run time adjusted to account for ramp-up time. New run time: %s' %
         timedelta(seconds=adjusted_time))
 
     ## Check if all required params are set
-    if not all([configData.homePath, configData.clientCertPath, configData.dbUri, configData.testHost, testFile]):
+    if not all([configData["homePath"], configData["clientCertPath"], configData["dbUri"], configData["testHost"], testFile]):
         print("Missing required arg (See -h for help on params)")
         sys.exit(2)
 
-    ## write out config file
+    ## write out to repository root config file (_NOT_ the file specified by "configPath")
     config.save(configData)
     setup.set_locust_test_name(testFile)