Sprint 196: Staging -> Master

APISubmissionGuide.md

@@ -6,24 +6,6 @@ While the Submission API has been designed to be as easy to understand as possib
 
 ## Login Process
 
-### Login to Max
-- Step 1: Authenticate with MAX directly to obtain the `ticket` value for Step 2
-    - Please refer to documentation provided by MAX.gov [here](./Using_Digital_Certificates_for_MAX_Authentication.pdf).
-    - Information about requesting Data Broker permissions within MAX can be found [here](https://community.max.gov/x/fJwuRQ).
-    - While we do not control MAX's login process, for simplicity purposes, here is a sample CURL request to the MAX login endpoint:
-    ```
-        curl -L -j -D - -b none 
-            --cert max.crt 
-            --key max.key 
-           https://serviceauth.max.gov/cas-cert/login?service=https://broker-api.usaspending.gov
-   ```
-
-- **NOTE**: Do **NOT** end the above service parameter url with a "/"
-- You would locate the `ticket` value in the `Location` header in the first header block returned by this request, i.e.,
-    `Location=https://broker-api.usaspending.gov?ticket=ST-123456-abcdefghijklmnopqrst-login.max.gov`
-- Step 2: call `/v1/max_login/` (POST) current broker login endpoint for logging into broker using MAX login. For details on its use, click [here](./doc/api_docs/login/max_login.md)
-    - Be sure to use the provided ticket within 30 seconds to ensure it does not expire.
-
 ### Login to the CAIA
 
 - Step 1: Get access to Treasury Mulesoft Exchange to view the Data Broker Experience API.
@@ -53,7 +35,8 @@ While the Submission API has been designed to be as easy to understand as possib
     - Each call to the Broker below in this guide (or any other endpoint referenced in the documentation) will be called slightly different:
         - The root url will be replaced:
             - Before: `https://broker-api.usaspending.gov/`
-            - After: `https://api.fiscal.treasury.gov/ap/prod/exp/v1/data-act-broker/`
+            - After: `https://api.fiscal.treasury.gov/ap/exp/v1/data-act-broker/`
+            - **NOTE**: The API Proxy does **not** support endpoint URLs ending with a trailing slash (`/`). Please ensure that all your calls to the API do not include a trailing slash.
         - Two new headers must be added in every request:
             - `client_id`: The `Client ID` copied from earlier.
             - `client_secret`: The `Client Secret` copied from earlier.
@@ -62,6 +45,8 @@ While the Submission API has been designed to be as easy to understand as possib
     - When finally logging into the Broker using the new credentials and you receive a message:
         - **"Authentication denied"**: The `client_id`/`client_secret` headers were not included. Make sure to include them.
         - **"Invalid Client"**: Your `client_id`/`client_secret` credentials were provided but incorrect. Ensure the values are correct.
+        - **"HTTP POST on resource '.../v1/proxy_login' failed: unauthorized (401)."**: Your system email was not initially entered into the database. Please reach out to the service desk with this error.
+        - **"resource_not_found"**: Your endpoint URL is ending with a slash which is currently not supported with the API proxy specifically. To resolve, simply remove the final slash.
 
 ## DABS Submission Process
 

dataactbroker/handlers/account_handler.py

@@ -3,7 +3,6 @@
 from operator import attrgetter
 import re
 import requests
-import xmltodict
 
 from flask import g
 
@@ -153,76 +152,6 @@ def proxy_login(self, session):
             # Return 500
             return JsonResponse.error(e, StatusCode.INTERNAL_ERROR)
 
-    def max_login(self, session):
-        """ Logs a user in if their password matches using MAX
-
-            Args:
-                session: Session object from flask
-
-            Returns:
-                A JsonResponse containing the user information or details on which error occurred, such as whether a
-                type was wrong, something wasn't implemented, invalid keys were provided, login was denied, or a
-                different, unexpected error occurred.
-        """
-        try:
-            safe_dictionary = RequestDictionary(self.request)
-
-            ticket = safe_dictionary.get_value("ticket")
-            service = safe_dictionary.get_value('service')
-
-            # Call MAX's serviceValidate endpoint and retrieve the response
-            max_dict = get_max_dict(ticket, service)
-
-            if 'cas:authenticationSuccess' not in max_dict['cas:serviceResponse']:
-                raise ValueError("The Max CAS endpoint was unable to locate your session "
-                                 "using the ticket/service combination you provided.")
-            cas_attrs = max_dict['cas:serviceResponse']['cas:authenticationSuccess']['cas:attributes']
-
-            # Grab MAX ID to see if a service account is being logged in
-            max_id_components = cas_attrs['cas:MAX-ID'].split('_')
-            service_account_flag = (len(max_id_components) > 1 and max_id_components[0].lower() == 's')
-
-            # Grab the email and list of groups from MAX's response
-            email = cas_attrs['cas:Email-Address']
-
-            try:
-                sess = GlobalDB.db().session
-                user = sess.query(User).filter(func.lower(User.email) == func.lower(email)).one_or_none()
-
-                # If the user does not exist, create them since they are allowed to access the site because they got
-                # past the above group membership checks
-                if user is None:
-                    user = User()
-                    user.email = email
-
-                first_name = cas_attrs['cas:First-Name']
-                middle_name = cas_attrs['cas:Middle-Name']
-                last_name = cas_attrs['cas:Last-Name']
-                set_user_name(user, first_name, middle_name, last_name)
-
-                set_max_perms(user, cas_attrs['cas:GroupList'], service_account_flag)
-
-                sess.add(user)
-                sess.commit()
-
-            except MultipleResultsFound:
-                raise ValueError("An error occurred during login.")
-
-            return self.create_session_and_response(session, user)
-
-        # Catch any specifically raised errors or any other errors that may have happened and return them cleanly.
-        # We add the error parameter here because this endpoint needs to provide better feedback, and to avoid changing
-        # the default behavior of the JsonResponse class globally.
-        except (TypeError, KeyError, NotImplementedError) as e:
-            # Return a 400 with appropriate message
-            return JsonResponse.error(e, StatusCode.CLIENT_ERROR, error=str(e))
-        except ValueError as e:
-            # Return a 401 for login denied
-            return JsonResponse.error(e, StatusCode.LOGIN_REQUIRED, error=str(e))
-        except Exception as e:
-            # Return 500
-            return JsonResponse.error(e, StatusCode.INTERNAL_ERROR, error=str(e))
-
     def caia_login(self, session):
         """ Logs a user in if their CAIA validation succeeds
 
@@ -376,8 +305,8 @@ def email_users(submission, system_email, template_type, user_ids):
         return JsonResponse.create(StatusCode.OK, {"message": "Emails successfully sent"})
 
 
-def perms_to_affiliations(perms, user_id, service_account_flag=False):
-    """ Convert a list of perms from MAX to a list of UserAffiliations. Filter out and log any malformed perms
+def perms_to_affiliations(perms, user_id):
+    """ Convert a list of perms from CAIA to a list of UserAffiliations. Filter out and log any malformed perms
 
         Args:
             perms: list of permissions (as lists [code, perm]) for the user
@@ -413,10 +342,7 @@ def perms_to_affiliations(perms, user_id, service_account_flag=False):
 
         perm_level = perm_level.lower()
 
-        if service_account_flag:
-            # Replace MAX Service Account permissions with Broker "write" and "editfabs" permissions
-            perm_level = 'we'
-        elif perm_level not in 'rwsef':
+        if perm_level not in 'rwsef':
             logger.warning(log_data)
             continue
 
@@ -466,7 +392,7 @@ def best_affiliation(affiliations):
 
 
 def set_user_name(user, first_name, middle_name, last_name):
-    """ Update the name for the user based on the MAX attributes.
+    """ Update the name for the user based on the CAIA attributes.
 
         Args:
             user: the User object
@@ -479,54 +405,6 @@ def set_user_name(user, first_name, middle_name, last_name):
         user.name = first_name + " " + middle_name[0] + ". " + last_name
 
 
-def set_max_perms(user, max_group_list, service_account_flag=False):
-    """ Convert the user group lists present on MAX into a list of UserAffiliations and/or website_admin status.
-
-        Permissions are encoded as a comma-separated list of:
-        {parent-group}-CGAC_{cgac-code}-PERM_{one-of-R-W-S-E-F}
-        {parent-group}-CGAC_{cgac-code}-FREC_{frec_code}-PERM_{one-of-R-W-S-E-F}
-        or
-        {parent-group}-CGAC_SYS to indicate website_admin
-
-        Args:
-            user: the User object
-            max_group_list: list of all MAX groups the user has
-            service_account_flag: flag to indicate a service account
-    """
-    prefix = CONFIG_BROKER['parent_group'] + '-CGAC_'
-
-    # Each group name that we care about begins with the prefix, but once we have that list, we don't need the
-    # prefix anymore, so trim it off.
-    if max_group_list is not None:
-        group_names = [group_name[len(prefix):]
-                       for group_name in max_group_list.split(',')
-                       if group_name.startswith(prefix)]
-    elif service_account_flag:
-        raise ValueError("There are no Data Broker permissions assigned to this Service Account. You may request "
-                         "permissions at https://community.max.gov/x/fJwuRQ")
-    else:
-        group_names = []
-
-    if 'SYS' in group_names:
-        user.website_admin = True
-        user.affiliations = []
-    else:
-        user.website_admin = False
-        perms = []
-        for group_name in group_names:
-            # Always starts with the 3-digit CGAC
-            code = group_name[:3]
-            if 'FREC' in group_name:
-                # If FREC, then its the [3-digit CGAC]-FREC_[4-digit FREC code]
-                code = group_name[9:13]
-            # Permission level is always the last character
-            perm = group_name[-1]
-            perms.append((code, perm))
-
-        if perms:
-            user.affiliations = best_affiliation(perms_to_affiliations(perms, user.user_id, service_account_flag))
-
-
 def set_caia_perms(user, roles):
     """ Convert the user group list present on CAIA into a list of UserAffiliations and/or website_admin status.
 
@@ -571,21 +449,6 @@ def json_for_user(user, session_id):
     }
 
 
-def get_max_dict(ticket, service):
-    """ Get the result from MAX's serviceValidate functionality
-
-        Args:
-            ticket: the ticket to send to MAX
-            service: the service to send to MAX
-
-        Returns:
-            A dictionary of the response from MAX
-    """
-    url = CONFIG_BROKER['cas_service_url'].format(ticket, service)
-    max_xml = requests.get(url).content
-    return xmltodict.parse(max_xml)
-
-
 def get_caia_tokens(code, redirect_uri):
     """ Verify the authorization code to get the logged in user's various tokens
 
@@ -637,7 +500,7 @@ def refresh_tokens(refresh_token, redirect_uri):
 
 
 def get_caia_user_dict(accces_token):
-    """ Get the result from MAX's serviceValidate functionality
+    """ Get the result from CAIA's serviceValidate functionality
 
         Args:
             accces_token: the access token of the logged in user

dataactbroker/routes/login_routes.py

@@ -1,5 +1,4 @@
 from flask import g, request, session
-from flask_deprecate import deprecate_route
 
 from dataactcore.utils.jsonResponse import JsonResponse
 from dataactcore.utils.statusCode import StatusCode
@@ -18,13 +17,6 @@ def proxy_login():
         account_manager = AccountHandler(request)
         return account_manager.proxy_login(session)
 
-    @app.route("/v1/max_login/", methods=["POST"])
-    @deprecate_route("MAX login will no longer be supported on January 1st, 2025."
-                     " Instead, CAIA login is supported now (via /v1/caia_login) and will be the only option then.")
-    def max_login():
-        account_manager = AccountHandler(request)
-        return account_manager.max_login(session)
-
     @app.route("/v1/caia_login/", methods=["POST"])
     def caia_login():
         account_manager = AccountHandler(request)

dataactcore/config_example.yml

@@ -94,17 +94,6 @@ broker:
     usas_public_reference_url: new-url.com
     usas_public_submissions_url: new-url.com
 
-    # link to help for failed MAX login
-    max_help_url: https://max_help_url.gov
-
-    # MAX login
-    cas_service_url: https://cas.service.url
-    parent_group: sample
-
-    max:
-        federal_hierarchy_api_url: https://federal.hierarchy.api.uri.gov/?api_key={}
-        federal_hierarchy_api_key: example
-
     # Session timeout, in seconds
     session_timeout: 1800
 

dataactcore/scripts/setup/load_location_data.py

@@ -220,6 +220,16 @@ def load_city_data(force_reload):
     with RetrieveFileFromUri(city_file_url, 'r').get_file_object() as city_file:
         new_data = parse_city_file(city_file)
 
+    # parse the legacy city code data
+    legacy_city_filename = 'NationalFedCodes_LEGACY.txt'
+    legacy_city_url = f'{CONFIG_BROKER["usas_public_reference_url"]}/{legacy_city_filename}'
+    with RetrieveFileFromUri(legacy_city_url, 'r').get_file_object() as legacy_city_file:
+        legacy_data = parse_city_file(legacy_city_file)
+
+    # only add legacy city code data if the state is not already in the new city data
+    add_legacy_data = legacy_data.loc[~legacy_data['state_code'].isin(new_data['state_code'].unique())]
+    new_data = pd.concat([new_data, add_legacy_data])
+
     diff_found = check_dataframe_diff(new_data, CityCode, ['city_code_id'], ['state_code', 'city_code'])
 
     if force_reload or diff_found:

doc/INSTALL.md

@@ -141,7 +141,7 @@ _:bulb: TIP: Many of the commands below use the format `docker exec -it dataact-
 ```bash
 $ docker exec -it dataact-broker-backend python dataactcore/scripts/initialize.py -a
 ```
-This creates a local admin user that you can use to log in. The Broker utilizes MAX.gov for login when using a remote server, but we cannot recieve their response locally so we use a username and password for local development login. The credentials for the user created are the values you have configured in the `db.admin_email` and `db.admin_password` config params in `config.yml` or overridden in `local_config.yml`.
+This creates a local admin user that you can use to log in. The Broker utilizes CAIA for login when using a remote server, but we cannot recieve their response locally so we use a username and password for local development login. The credentials for the user created are the values you have configured in the `db.admin_email` and `db.admin_password` config params in `config.yml` or overridden in `local_config.yml`.
 
 Now try to browse to http://localhost:3002, and login with the configured credentials (`[email protected]` / `password`). You should get past the login screen to the home screen.
 

doc/README.md

@@ -135,7 +135,6 @@ Login routes are used to log a user in or out or check if the current session is
 
 - [login](./api_docs/login/login.md)
 - [logout](./api_docs/login/logout.md)
-- [max\_login](./api_docs/login/max_login.md) (Deprecated, use `caia_login` instead)
 - [caia\_login](./api_docs/login/caia_login.md)
 - [session](./api_docs/login/session.md)
 

doc/api_docs/login/login.md

@@ -2,7 +2,7 @@
 
 **THIS ENDPOINT IS FOR LOCAL DEVELOPMENT ONLY AND CANNOT BE USED TO AUTHENTICATE INTO BROKER IN PRODUCTION**
 
-This route checks the username and password against a credentials file. It is used solely as a workaround for developing on a local instance of the broker to bypass MAX.gov login. Accepts input as json or form-urlencoded, with keys "username" and "password". See `active_user` docs for details.
+This route checks the username and password against a credentials file. It is used solely as a workaround for developing on a local instance of the broker to bypass CAIA login. Accepts input as json or form-urlencoded, with keys "username" and "password". See `active_user` docs for details.
 
 ## Body (JSON)