Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UX bug #18

Open
5 tasks
0kl-usds opened this issue Nov 30, 2021 · 0 comments
Open
5 tasks

UX bug #18

0kl-usds opened this issue Nov 30, 2021 · 0 comments

Comments

@0kl-usds
Copy link
Collaborator

From the bottom:

Remediation

  • Only allow one browser tab open at a time (same machine multiple tabs)
  • Timeout unused app instances (UX)
  • Only allow one session per a user at a time (different machines)
  • Only allow evaluation if queue is active

From Hypothesis 1:

  • When applicants are complete set all existing queue assignments members to false

Conclusion

There is no systemic risk of the system being faulty. Applicant DaF should have
the evaluation by SME_one invalidated, and a fuzzy checker should be run over
all hiring action to see if there are similar cases of SMEs using the tool in multiple tabs.

  1. An advisory should be sent out to SMEs to keep a clean desktop environment and work
    in only one tab. Alternatively, refreshing the page if you are unsure if you have multiple tabs
    will also ensure that you are in the correct space.
  2. An advisory should be sent out to SMEs to doublecheck the name matches before submitting
    an evaluation.
  3. A UX fix should go out to force the applicant name to be shown at all times to SMES.
  4. A bug fix should go out to prevent multiple browser tabs from maintaining different states.
  5. A bug fix should go out to prevent multiple user sessions from being valid.

SME names have been expunged and applicant names have been reduced to initials.
The two applicants of interest have been changed to JoH and DaF to make reading
and differentiating them easier.

Background - SME sees same applicant 3x times.

SME_one reported duplicate evaluations of an applicant, JoH and being presented
with the applicant 3x times.

While investigating this issue we found that there were no duplicate evaluations
of JoH, but that a review meant for JoH was actually applied to DaF.

To find out how such assignment we pulled the application_assignment table
for the SME_one, where rows are created in the table whenever a new assignment is
made:

SELECT
    a.name
    , a.id
    , asgn.created_at assignment_creation_date
    , asgn.updated_at assignment_update_date
    , expires
    FROM application_assignments as asgn
    LEFT JOIN app_user au ON au.id = asgn.evaluator_id
    LEFT JOIN applicant a ON a.id = asgn.applicant_id
    WHERE email='SME_one';

Which is displayed below:

name id assignment_creation_date assignment_update_date expires
ASGN:JR 6d1fe480-4bf6-4f4a-aeb6-a18e94755391 2021-11-23 21:21:00.499166 2021-11-23 21:29:36.715729 2021-11-23 21:21:00.499166
ASGN:DaF 6ce242a1-fde5-430f-b3da-dceb15c87ae9 2021-11-23 21:29:40.796335 2021-11-24 13:48:02.896608 2021-11-23 21:29:40.796335
ASGN:AK 6281f605-f322-44f2-be8a-d77534987d16 2021-11-24 13:35:54.345595 2021-11-24 13:37:40.199799 2021-11-24 13:35:54.345595
ASGN:JoH 602b1f78-6a01-49f6-a16a-5cfd1706fae5 2021-11-24 13:37:42.11964 2021-11-24 14:20:04.844465 2021-11-24 13:37:42.11964
ASGN:KS 5de4e9ad-58b7-4d19-9245-4d72b2fbd75e 2021-11-24 14:20:06.794886 2021-11-24 14:23:02.638973 2021-11-24 14:20:06.794886

As we can see, DaF was assigned before JoH, so an evaluation of JoH going to DaF
by system error would be extremely alarming.

Based on how the system works, this should be impossible under "normal"
operating conditions since evaluations are submitted with the UUID of the applicant
that is currently being displayed (i.e. there would have to be a custom server
request made by the SME_one to be able to send an evaluation meant for the applicant they
are evaluating to a different one - which SME_one's would have no incentive to do since
it is their evaluation either way.)

On interview with the SME_one while trying to find the cause of the error, it was
discovered that the SME_one had multiple browsers open that they were not aware of.

This led us to consider that it might be a human error, though we would need
confirmation because if it wasn't human error then there would be significant
issues with continuing to utilize the tool.

Hypothesis 1: Stale queue

The first hypothesis is that due to multiple windows, the following took place:

  1. SME_one opens window_one to do evaluations on 2021-11-23
  2. at 2021-11-23 21:29:40 the SME_one is assigned DaF in window_one
  3. SME_one signs off for the day without closing window_one
  4. the next day, the SME_one resumes reviewing, but opens a new window_two
  5. SME_one is assigned and evaluates AK in window_two
    • This means that DaF will have had to have been "completed" in order for SME_one to be assigned a new applicant.
  6. SME_one is assigned JoH in window_two [1st JoH]
  7. SME_one accidentally changes to window_one while looking at JoH's resume
  8. SME_one writes and submits JoH's evaluation in window_one, where the active applicant is DaF
  9. window_one pulls JoH from the queue and the applicant is presented with JoH again [2nd JoH]
  10. SME_one complete's review of JoH and submits it without error in window_one
  11. At some point SME_one returns to window_two and see's JoH again [3rd JoH]
  12. Flags are raised.

If that is the case we would expect to see the following in the database to confirm:

  1. SME_one assignment to DaF is before SME_one assignment to JoH
  2. DaF evaluations completed by 2 other SMEs before SME_one assignment to JoH[^1]
  3. DaF and JoH evaluations have no evaluations between them[2^]
    • Unless there are 2 other SME evaluations before SME_one's evaluation of JoH

[1^]: This is the only way that SME_one would be able to get a new applicant,
as if the applicant was not yet complete, SME_one would not be able to be
assigned a new applicant.

[2^]: There should be no way for the SMEone to review _another applicant before JoH,
unless we repeat what happened with DaF.

As we already have assignment data, what we need is the application evaluation logs
for JoH and DaF with SME_one identified. This will either prove or disprove
the hypothesis.

SELECT
    au.name evaluator
    , app.name applicant
    , ae.created_at
    , applicant_id
    FROM application_evaluation ae
    LEFT JOIN application a ON a.id = ae.application_id
    LEFT JOIN app_user au ON au.id = ae.evaluator
    LEFT JOIN applicant app ON app.id = applicant_id
    WHERE applicant_id = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'
    OR applicant_id ='602b1f78-6a01-49f6-a16a-5cfd1706fae5'
    OR applicant_id ='6281f605-f322-44f2-be8a-d77534987d16'
    ORDER BY ae.created_at;

Evaluation history

evaluator applicant created_at applicant_id
Eval: SME_three AK 2021-11-23 22:13:24.319139 6281f605-f322-44f2-be8a-d77534987d16
Eval: SME_two JoH 2021-11-23 22:37:22.043539 602b1f78-6a01-49f6-a16a-5cfd1706fae5
Eval: SME_one DaF 2021-11-24 13:35:52.754871 6ce242a1-fde5-430f-b3da-dceb15c87ae9
Eval: SME_one AK 2021-11-24 13:37:40.199799 6281f605-f322-44f2-be8a-d77534987d16
Eval: SME_one JoH 2021-11-24 14:20:04.844465 602b1f78-6a01-49f6-a16a-5cfd1706fae5
Eval: SME_two DaF 2021-11-24 17:06:53.484374 6ce242a1-fde5-430f-b3da-dceb15c87ae9
Eval: SME_three DaF 2021-11-24 17:47:09.018914 6ce242a1-fde5-430f-b3da-dceb15c87ae9
Eval: SME_three JoH 2021-11-24 18:17:33.792375 602b1f78-6a01-49f6-a16a-5cfd1706fae5

Hypothesis 1 invalidated: SMEtwo and SME_three evaluations followed _after SME_one's evaluation

Hypothesis 1 Mitigation:

  • When applicants are complete set all existing queue assignments members to false
  • Only allow evaluation if queue is active

Hypothesis 2 - Multi tabs

With that hypothesis invalidated, we then pulled all the assignment data for SME_one
and interleaved it with evaluation data for the three SMEs starting wtih DaF and
ending with JoH to see if there were any noticeable patterns[^3]:

evaluator applicant created_at assignment_update_timestamp - UPDT
1 ASGN: SME_one JR 2021-11-23 21:21:00.499166 2021-11-23 21:29:36.715729
2 ASGN: SME_one DaF 2021-11-23 21:29:40.796335 2021-11-24 13:48:02.896608*
3 Eval: SME_three AK 2021-11-23 22:13:24.319139
4 Eval: SME_two JoH 2021-11-23 22:37:22.043539
5 Eval: SME_one DaF 2021-11-24 13:35:52.754871
6 ASGN: SME_one AK 2021-11-24 13:35:54.345595 2021-11-24 13:37:40.199799
7 Eval: SME_one AK 2021-11-24 13:37:40.199799
8 ASGN: SME_one JoH 2021-11-24 13:37:42.11964 * 2021-11-24 14:20:04.844465
9 Eval: SME_one JoH 2021-11-24 14:20:04.844465*
10 ASGN: SME_one KS 2021-11-24 14:20:06.794886 2021-11-24 14:23:02.638973
11 Eval: SME_two DaF 2021-11-24 17:06:53.484374
12 Eval: SME_three DaF 2021-11-24 17:47:09.018914
13 Eval: SME_three JoH 2021-11-24 18:17:33.792375

From this we can see the usual pattern (see row 5) of evaluation creation and the
next assignment creation being within a second of each other:

-  5 EVAL: 13:35:52 - DAF
   6 ASGN: 13:35:54 - AK
-  7 EVAL: 13:37:40 - AK
   8 ASGN: 13:37:42 - JoH
-  9 EVAL: 14:20:04 - JoH
  10 ASGN: 14:20:06 - KS

The other interesting pattern is that the assignment update also correlates to the assignment:

-  6 UPDT: 13:37:40.199799 - AK
   7 EVAL: 13:37:40.199799 - AK
-  8 UPDT: 14:20:04.844465 - JOH
   9 EVAL: 14:20:04.844465 - JOH

But there is a significant offset for DaF's update, especially since it is after
DaF's created date, which is very unusual since the update and evaluation
are in the same database transaction.

2 UPDT: 13:48:02.896608 - DAF
5 EVAL: 13:35:52.754871 - DAF

This generates a new hypothesis:

  1. SME_one opens window_one to do evaluations on 2021-11-23
  2. at 2021-11-23 21:29:40 the SME_one is assigned DaF in window_one
  3. SME_one signs off for the day without closing window_one
  4. the next day, the SME_one resumes reviewing, but opens a new window_two
  5. SME_one is presented with DaF in window_two, and evaluates candidate
  6. SME_one is then assigned and evaluates AK in window_two
  7. SME_one is then assigned and sees JoH, and while changing between windows
    and resume changes back to window_one, but does not see that the current
    window is assigned to DaF [1st JoH]
  8. SME_one places review into DaF's evaluation and submits it
    • This overwrites the previous evaluation and causes an update (2 UPDT)
  9. Since SME_one is still assigned JoH, they are presented with JoH in window_one [2nd JoH]
  10. SME_one then evaluates JoH in window_one - this time correctly so
  11. At some point SME_one changes back to window_two and sees JoH again [3rd Joh]
  12. Flags are raised.

If this is the case, we would see the example behavior as above (that the assignment
update time and the eval creation time are grossly out of sync - i.e. that the
assignment was updated after the evaluation was completed).

More than that, we would expect several entries in our audit log for DaF.

Pulling those out:

SELECT id
, name
, email
, hr_review_time
, approved
, evaluation_feedback
, passing
FROM application_evaluation_audit_trail
WHERE assessment_hurdle_id = '7b8d1165-04e4-4b86-8747-137249dc825d'
AND name LIKE '%DaF%';

Which gives us this:

id name email hr_review_time approved evaluation_feedback passing
543b9458-4f5e-4bff-8b31-a86d095b23b4 DaF SME_one 2021-11-29 21:43:34.212177 t t
c3eea0b4-bd34-4d88-8b14-7641ccbd981f DaF SME_two 2021-11-29 21:44:28.402855 t f
3bed613e-fbfe-49c6-9d5c-60aa09d52e2c DaF SME_three 2021-11-29 21:45:04.372087 t f

Clearly not what we're after, the audit trail tracks only on HR actions.

So let's go to our application logs... which we have timestamps to search:
First we need to verify that the initial created at date for the evaluation had a real evaluation made:

  1. 2021-11-24 13:35:52.754871

Then we need to check for a second, erroneous evaluation at the assignment updated time:

  1. 2021-11-24 13:48:02.896608

If these both have two separate evaluations, then it is highly likely that this
is the order of events that transpired.

For Nov 24th there are 7 records of with both SME_one's id (99f4a68e) and DaFs
id (6ce242a1).
They are as follows:

Note Time @cf.org @cf.space @cf.app @message
Returning user from queue on new login Nov 24, 2021 @ 08:32:40.415 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Existing Active item in Queue for user: 99f4a68e-5720-4317-9127-9122906470b0 returning existing item 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:32:40"}
Get applicant Information Nov 24, 2021 @ 08:32:40.701 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_feedback", "applicant_id", "evaluator_id", "feedback_author_id", "feedback_timestamp", "created_at", "updated_at" FROM "public"."applicant_evaluation_feedback" AS "ApplicantEvaluationFeedback" WHERE "ApplicantEvaluationFeedback"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9' AND "ApplicantEvaluationFeedback"."evaluator_id" = '99f4a68e-5720-4317-9127-9122906470b0' LIMIT 1; {"timestamp":"2021-11-24 08:32:40"}
CompetencyEvaluation selection Nov 24, 2021 @ 08:32:40.709 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:32:40"}
CompetencyEvaluation selection Nov 24, 2021 @ 08:35:52.831 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
Update assignment to inactive Nov 24, 2021 @ 08:35:52.844 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: [Transaction] Updated evaluator [99f4a68e-5720-4317-9127-9122906470b0] assignment of 6ce242a1-fde5-430f-b3da-dceb15c87ae9 to inactive {"timestamp":"2021-11-24 08:35:52"}
CompetencyEvaluation selection Nov 24, 2021 @ 08:48:02.973 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
Update assignment to inactive Nov 24, 2021 @ 08:48:02.987 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: [Transaction] Updated evaluator [99f4a68e-5720-4317-9127-9122906470b0] assignment of 6ce242a1-fde5-430f-b3da-dceb15c87ae9 to inactive {"timestamp":"2021-11-24 08:48:02"}

We can now move entirely to the "log time" instead of our "audit time" since we
can make the "update assignment to inactive" it our own log events confirming that we
twice updated the evaluator's assignment to inactive. We can now search the logs
around these timestamps for duplicate evaluations: specifically around 8:35 and 8:4
while searching only for the applicant by their id (6ce242a1):

Note Time @cf.org @cf.space @cf.app @message
1 Nov 24, 2021 @ 08:32:40.415 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Existing Active item in Queue for user: 99f4a68e-5720-4317-9127-9122906470b0 returning existing item 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:32:40"}
2 Nov 24, 2021 @ 08:32:40.700 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "name", "flag_type", "flag_message", "assessment_hurdle_id", "additional_note", "created_at", "updated_at" FROM "public"."applicant" AS "Applicant" WHERE "Applicant"."id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:32:40"}
3 Nov 24, 2021 @ 08:32:40.701 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_feedback", "applicant_id", "evaluator_id", "feedback_author_id", "feedback_timestamp", "created_at", "updated_at" FROM "public"."applicant_evaluation_feedback" AS "ApplicantEvaluationFeedback" WHERE "ApplicantEvaluationFeedback"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9' AND "ApplicantEvaluationFeedback"."evaluator_id" = '99f4a68e-5720-4317-9127-9122906470b0' LIMIT 1; {"timestamp":"2021-11-24 08:32:40"}
4 Nov 24, 2021 @ 08:32:40.704 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "Application"."id", "Application"."applicant_id", "Application"."specialty_id", "Application"."created_at", "Application"."updated_at", "Specialty"."id" AS "Specialty.id" FROM "public"."application" AS "Application" INNER JOIN "public"."specialty" AS "Specialty" ON "Application"."specialty_id" = "Specialty"."id" WHERE "Application"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:32:40"}
5 Nov 24, 2021 @ 08:32:40.709 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:32:40"}
6 Nov 24, 2021 @ 08:32:40.710 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "competency_id", "does_not_meet", "meets" FROM "public"."competency_evaluation_count" AS "CompetencyEvaluationCount" WHERE "CompetencyEvaluationCount"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:32:40"}
7 Nov 24, 2021 @ 08:32:40.713 gsa-open-opportunities USDS smeqa-rr [32minfo[39m: 127.0.0.1 - - [24/Nov/2021:13:32:40 +0000] "GET /api/evaluation/7b8d1165-04e4-4b86-8747-137249dc825d/display/6ce242a1-fde5-430f-b3da-dceb15c87ae9 HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" {"timestamp":"2021-11-24 08:32:40"}
8 Evaluation submission Nov 24, 2021 @ 08:35:52.819 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Submit Evaluation for 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:35:52"}
9 Nov 24, 2021 @ 08:35:52.820 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "name", "flag_type", "flag_message", "assessment_hurdle_id", "additional_note", "created_at", "updated_at" FROM "public"."applicant" AS "Applicant" WHERE "Applicant"."id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
10 Nov 24, 2021 @ 08:35:52.822 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "Application"."id", "Application"."applicant_id", "Application"."specialty_id", "Application"."created_at", "Application"."updated_at", "Specialty"."id" AS "Specialty.id", "Specialty"."name" AS "Specialty.name", "Specialty"."local_id" AS "Specialty.local_id", "Specialty"."assessment_hurdle_id" AS "Specialty.assessment_hurdle_id", "Specialty"."points_required" AS "Specialty.points_required", "Specialty"."created_at" AS "Specialty.created_at", "Specialty"."updated_at" AS "Specialty.updated_at", "Specialty->SpecialtyCompetencies"."id" AS "Specialty.SpecialtyCompetencies.id", "Specialty->SpecialtyCompetencies"."specialty_id" AS "Specialty.SpecialtyCompetencies.specialty_id", "Specialty->SpecialtyCompetencies"."competency_id" AS "Specialty.SpecialtyCompetencies.competency_id", "Specialty->SpecialtyCompetencies"."created_at" AS "Specialty.SpecialtyCompetencies.created_at", "Specialty->SpecialtyCompetencies"."updated_at" AS "Specialty.SpecialtyCompetencies.updated_at" FROM "public"."application" AS "Application" INNER JOIN "public"."specialty" AS "Specialty" ON "Application"."specialty_id" = "Specialty"."id" LEFT OUTER JOIN "public"."specialty_competencies" AS "Specialty->SpecialtyCompetencies" ON "Specialty"."id" = "Specialty->SpecialtyCompetencies"."specialty_id" WHERE "Application"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
11 Nov 24, 2021 @ 08:35:52.827 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "Application"."id", "Application"."applicant_id", "Application"."specialty_id", "Application"."created_at", "Application"."updated_at", "Specialty"."id" AS "Specialty.id" FROM "public"."application" AS "Application" INNER JOIN "public"."specialty" AS "Specialty" ON "Application"."specialty_id" = "Specialty"."id" WHERE "Application"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
12 Nov 24, 2021 @ 08:35:52.831 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
13 Nov 24, 2021 @ 08:35:52.832 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "competency_id", "does_not_meet", "meets" FROM "public"."competency_evaluation_count" AS "CompetencyEvaluationCount" WHERE "CompetencyEvaluationCount"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:35:52"}
14 Nov 24, 2021 @ 08:35:52.833 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Creating evaluation results for 1 applications for applicant 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:35:52"}
15 Nov 24, 2021 @ 08:35:52.844 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: [Transaction] Updated evaluator [99f4a68e-5720-4317-9127-9122906470b0] assignment of 6ce242a1-fde5-430f-b3da-dceb15c87ae9 to inactive {"timestamp":"2021-11-24 08:35:52"}
16 Nov 24, 2021 @ 08:35:52.850 gsa-open-opportunities USDS smeqa-rr [32minfo[39m: 127.0.0.1 - - [24/Nov/2021:13:35:52 +0000] "PUT /api/evaluation/7b8d1165-04e4-4b86-8747-137249dc825d/submit/6ce242a1-fde5-430f-b3da-dceb15c87ae9 HTTP/1.1" 201 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" {"timestamp":"2021-11-24 08:35:52"}
17 Nov 24, 2021 @ 08:35:54.423 gsa-open-opportunities USDS smeqa-rr AND active IS true) AS "current_queue_count" FROM "public"."applicant_queue" AS "ApplicantQueue" WHERE "ApplicantQueue"."assessment_hurdle_id" = '7b8d1165-04e4-4b86-8747-137249dc825d' AND "ApplicantQueue"."applicant_id" NOT IN ('ef0e42c5-17cf-4b06-a5ea-f5965c1f0858', '91a92152-074c-479e-8e7f-03be8c6e00c7', '7c29b42a-e2bf-4fbb-89d3-8bf1ed67dfa7', '7949f930-ef71-4248-b349-6aad67318c2c', '7831ec99-7ce7-4628-9d62-7e995b2c426d', '77c23896-66c7-4543-80bb-8920becff7cc', '779dfaa3-a0dd-4dc7-8670-1b63a3a5e060', '7679075a-913c-4291-a590-a8a659492194', '74a526dd-40ed-4b02-a70c-a5c887ac8e67', '6fb6b56c-ed15-45ed-881c-888f78093d7c', '6d8130f6-c2e5-4a59-bf8b-70c82337fa23', '6d1fe480-4bf6-4f4a-aeb6-a18e94755391', '6ce242a1-fde5-430f-b3da-dceb15c87ae9', '88414f59-43af-4e90-a5c7-95f30f86e3a7', 'fcc2e69f-49aa-4dfc-bb87-a5110f32aac0', '6f5a351e-71b7-426f-82f4-34f595bf616f', '646bbf9f-d1c6-4afa-83bb-8f441bb6d86e') GROUP BY "ApplicantQueue"."applicant_id", "evaluators" ORDER BY current_queue_count ASC NULLS FIRST, evaluators DESC NULLS LAST, applicant_id DESC LIMIT 1; {"timestamp":"2021-11-24 08:35:54"}
18 Nov 24, 2021 @ 08:37:42.108 gsa-open-opportunities USDS smeqa-rr AND active IS true) AS "current_queue_count" FROM "public"."applicant_queue" AS "ApplicantQueue" WHERE "ApplicantQueue"."assessment_hurdle_id" = '7b8d1165-04e4-4b86-8747-137249dc825d' AND "ApplicantQueue"."applicant_id" NOT IN ('ef0e42c5-17cf-4b06-a5ea-f5965c1f0858', '91a92152-074c-479e-8e7f-03be8c6e00c7', '7c29b42a-e2bf-4fbb-89d3-8bf1ed67dfa7', '7949f930-ef71-4248-b349-6aad67318c2c', '7831ec99-7ce7-4628-9d62-7e995b2c426d', '77c23896-66c7-4543-80bb-8920becff7cc', '779dfaa3-a0dd-4dc7-8670-1b63a3a5e060', '7679075a-913c-4291-a590-a8a659492194', '74a526dd-40ed-4b02-a70c-a5c887ac8e67', '6fb6b56c-ed15-45ed-881c-888f78093d7c', '6d8130f6-c2e5-4a59-bf8b-70c82337fa23', '6d1fe480-4bf6-4f4a-aeb6-a18e94755391', '6ce242a1-fde5-430f-b3da-dceb15c87ae9', '6281f605-f322-44f2-be8a-d77534987d16', '88414f59-43af-4e90-a5c7-95f30f86e3a7', 'fcc2e69f-49aa-4dfc-bb87-a5110f32aac0', '6f5a351e-71b7-426f-82f4-34f595bf616f', '646bbf9f-d1c6-4afa-83bb-8f441bb6d86e') GROUP BY "ApplicantQueue"."applicant_id", "evaluators" ORDER BY current_queue_count ASC NULLS FIRST, evaluators DESC NULLS LAST, applicant_id DESC LIMIT 1; {"timestamp":"2021-11-24 08:37:42"}
19 Evaluation Submission Nov 24, 2021 @ 08:48:02.961 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Submit Evaluation for 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:48:02"}
20 Nov 24, 2021 @ 08:48:02.962 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "name", "flag_type", "flag_message", "assessment_hurdle_id", "additional_note", "created_at", "updated_at" FROM "public"."applicant" AS "Applicant" WHERE "Applicant"."id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
21 Nov 24, 2021 @ 08:48:02.965 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 2ms Executed (default): SELECT "Application"."id", "Application"."applicant_id", "Application"."specialty_id", "Application"."created_at", "Application"."updated_at", "Specialty"."id" AS "Specialty.id", "Specialty"."name" AS "Specialty.name", "Specialty"."local_id" AS "Specialty.local_id", "Specialty"."assessment_hurdle_id" AS "Specialty.assessment_hurdle_id", "Specialty"."points_required" AS "Specialty.points_required", "Specialty"."created_at" AS "Specialty.created_at", "Specialty"."updated_at" AS "Specialty.updated_at", "Specialty->SpecialtyCompetencies"."id" AS "Specialty.SpecialtyCompetencies.id", "Specialty->SpecialtyCompetencies"."specialty_id" AS "Specialty.SpecialtyCompetencies.specialty_id", "Specialty->SpecialtyCompetencies"."competency_id" AS "Specialty.SpecialtyCompetencies.competency_id", "Specialty->SpecialtyCompetencies"."created_at" AS "Specialty.SpecialtyCompetencies.created_at", "Specialty->SpecialtyCompetencies"."updated_at" AS "Specialty.SpecialtyCompetencies.updated_at" FROM "public"."application" AS "Application" INNER JOIN "public"."specialty" AS "Specialty" ON "Application"."specialty_id" = "Specialty"."id" LEFT OUTER JOIN "public"."specialty_competencies" AS "Specialty->SpecialtyCompetencies" ON "Specialty"."id" = "Specialty->SpecialtyCompetencies"."specialty_id" WHERE "Application"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
22 Nov 24, 2021 @ 08:48:02.969 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "Application"."id", "Application"."applicant_id", "Application"."specialty_id", "Application"."created_at", "Application"."updated_at", "Specialty"."id" AS "Specialty.id" FROM "public"."application" AS "Application" INNER JOIN "public"."specialty" AS "Specialty" ON "Application"."specialty_id" = "Specialty"."id" WHERE "Application"."applicant_id" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
23 Nov 24, 2021 @ 08:48:02.973 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "id", "evaluation_note", "evaluator", "applicant", "competency_id", "competency_selector_id", "created_at", "updated_at" FROM "public"."competency_evaluation" AS "CompetencyEvaluation" WHERE "CompetencyEvaluation"."competency_id" IN ('e4316443-7be8-40ab-b913-9dfc4c6b83b8', '984a2d5f-5d9d-426f-bf51-486d1bc42570') AND "CompetencyEvaluation"."evaluator" = '99f4a68e-5720-4317-9127-9122906470b0' AND "CompetencyEvaluation"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
24 Nov 24, 2021 @ 08:48:02.975 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: 1ms Executed (default): SELECT "competency_id", "does_not_meet", "meets" FROM "public"."competency_evaluation_count" AS "CompetencyEvaluationCount" WHERE "CompetencyEvaluationCount"."applicant" = '6ce242a1-fde5-430f-b3da-dceb15c87ae9'; {"timestamp":"2021-11-24 08:48:02"}
25 Nov 24, 2021 @ 08:48:02.975 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: Creating evaluation results for 1 applications for applicant 6ce242a1-fde5-430f-b3da-dceb15c87ae9 {"timestamp":"2021-11-24 08:48:02"}
26 Nov 24, 2021 @ 08:48:02.987 gsa-open-opportunities USDS smeqa-rr [34mdebug[39m: [Transaction] Updated evaluator [99f4a68e-5720-4317-9127-9122906470b0] assignment of 6ce242a1-fde5-430f-b3da-dceb15c87ae9 to inactive {"timestamp":"2021-11-24 08:48:02"}
27 Nov 24, 2021 @ 08:48:02.992 gsa-open-opportunities USDS smeqa-rr [32minfo[39m: 127.0.0.1 - - [24/Nov/2021:13:48:02 +0000] "PUT /api/evaluation/7b8d1165-04e4-4b86-8747-137249dc825d/submit/6ce242a1-fde5-430f-b3da-dceb15c87ae9 HTTP/1.1" 201 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" {"timestamp":"2021-11-24 08:48:02"}`

It is worth nothing that all other evaluation submissions from other SMEs
happened >4 hours after the time window being examined, so all evaluations
would be from SME_one.

From the applications logs we see two evaluations: - Line (8) @ 8:32:52 - Line (19) @ 8:48:02

From the audit logs we have remarkable similar timestamps for the creation of the
initial evaluation and update of the assignments (we have to use the "updated at" of the
assignment as a proxy for the "evaluation update time" because actions taken on
the "evaluation" will update the "updated at" value ): - 2 ASGN: Updated at: 13:48:02 - 5 EVAL: 13:34:52

Hypothesis 2 validated: It's clear from the matching timestamps that the timezones are off by 5 hours, but otherwise
these are the same transactions. This shows that SME_one submitted twice not on
JoH as expected, but actually on DaF due to multiple browser tabs being open.

Hypothesis 2 Mitigation:

  • Only allow one browser tab open at a time (same machine multiple tabs)
  • Timeout unused browser tabs (UX)
  • Only allow one session per a user at a time (different machines)
  • Only allow evaluation if queue is active

Root cause

Why did the SME submit an evaluation on the wrong applicant?
Application did not prevent multiple browser tabs from being open
Application allowed for multiple submissions on the same applicant (this is expected behavior).

Why didn't the application prevent multiple browser tabs from being open?
Unexpected use case, browser does not time out.

Root cause here is missing UX regarding common usage with multiple tabs/browsers.

Remediation

  • Only allow one browser tab open at a time (same machine multiple tabs)
  • Timeout unused browser tabs (UX)
  • Only allow one session per a user at a time (different machines)
  • Only allow evaluation if queue is active

From Hypothesis 1:

  • When applicants are complete set all existing queue assignments members to false

Conclusion

There is no systemic risk of the system being faulty. Applicant DaF should have
the evaluation by SME_one invalidated, and a fuzzy checker should be run over
all hiring action to see if there are similar cases of SMEs using the tool in multiple tabs.

  1. An advisory should be sent out to SMEs to keep a clean desktop environment and work
    in only one tab. Alternatively, refreshing the page if you are unsure if you have multiple tabs
    will also ensure that you are in the correct space.
  2. An advisory should be sent out to SMEs to doublecheck the name matches before submitting
    an evaluation.
  3. A UX fix should go out to force the applicant name to be shown at all times to SMES.
  4. A bug fix should go out to prevent multiple browser tabs from maintaining different states.
  5. A bug fix should go out to prevent multiple user sessions from being valid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@0kl-usds