feat: SSO Improvement - alter user_sessions table to include access token, implement CRUD ops, GET, POST, PATCH APIs and det token CLIs

[ShreyaLnuHpe]

Ticket

This is a feature branch for SSO Improvement project
DET-10397
DET-10396
DET-10454
DET-10403
DET-10455
DET-10398
DET-10405
DET-10455

Description

Allowing users to create long lived access tokens that they can use for authentication.

Altering 'user_sessions' table for tracking access tokens that can be used for authentication & association with the appropriate user.

Updated table fields:

token_type AS ENUM (
'USER_SESSION', 
'ACCESS_TOKEN'
);

public.user_sessions (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    user_id integer NOT NULL,
    expiry TIMESTAMP NOT NULL,
    created_at TIMESTAMP DEFAULT NULL,
    token_type token_type NOT NULL DEFAULT 'USER_SESSION',
    revoked_at TIMESTAMP DEFAULT NULL,
    description TEXT DEFAULT NULL,
)

APIs:

POST /api/v1/tokens - Create and get a user's access token.
GET /api/v1/tokens - Get a list of all access token records.
PATCH /api/v1/tokens/{token_id} - Patch an access token's mutable fields.

CLI commands:

1. det token -h

% det token -h
usage: det token [-h] subcommand ...

positional arguments:
  subcommand
    help      show help for this command
    create    create token
    describe  describe token info
    edit      edit token info
    list (ls)
              list all active access tokens
    login     log in with token
    revoke    revoke token

optional arguments:
  -h, --help  show this help message and exit

2. det token login [-h] token

% det token login v2.public.eyJpZCI6Mzk2LCJ1c2VyX2lkIjoyLCJleHBpcnkiOiIyMDI0LTEwLTEyVDIzOjQzOjI5LjY5NjU0N1oiLCJjcmVhdGVkX2F0IjoiMjAyNC0xMC0xMVQxNzo0MzoyOS42OTY1NDdaIiwidG9rZW5fdHlwZSI6IkFDQ0VTU19UT0tFTiIsInJldm9rZWQiOmZhbHNlLCJkZXNjcmlwdGlvbiI6InRlc3QgdG9rZW4iLCJJbmhlcml0ZWRDbGFpbXMiOm51bGx9lXy_vF5P1p3X5RtEx6VOzek6AsgVjul4K8jeYbuLBWlYYmng49JERbP8r67MKSj7Ol6CCDzd6z5OI_-yoaaCCg.bnVsbA

Authenticated as determined.

3. det token list [-h] [--show-inactive] [--json | --yaml] [username]

% det token ls                                                                  

   ID |   User ID | Description      | Created At                  | Expires At                  | Revoked   | Token Type
------+-----------+------------------+-----------------------------+-----------------------------+-----------+------------------
 5656 |      2305 |                  | 2024-09-18T05:12:48.424089Z | 2024-09-18T05:12:50.424089Z | True      | ACCESS_TOKEN
 5662 |         1 |                  | 2024-09-18T05:41:55.342039Z | 2024-09-18T05:41:57.342039Z | True      | ACCESS_TOKEN
 5673 |      2305 |                  | 2024-09-18T17:50:02.784704Z | 2024-09-18T17:50:04.784704Z | False     | ACCESS_TOKEN
 5675 |         1 |                  | 2024-09-18T17:59:52.688643Z | 2024-09-18T17:59:54.688643Z | False     | ACCESS_TOKEN
...

4. det token create [-h] [--expiration-days EXPIRATION_DAYS] [--description DESCRIPTION] [--json | --yaml] [username]

 % det token create determined -e 30 -d "test token" --yaml
TokenId: 397

Access Token: 
v2.public.eyJpZCI6Mzk3LCJ1c2VyX2lkIjoyLCJleHBpcnkiOiIyMDI0LTEwLTEyVDIzOjQzOjM4LjI2NTkxN1oiLCJjcmVhdGVkX2F0IjoiMjAyNC0xMC0xMVQxNzo0MzozOC4yNjU5MTdaIiwidG9rZW5fdHlwZSI6IkFDQ0VTU19UT0tFTiIsInJldm9rZWQiOmZhbHNlLCJkZXNjcmlwdGlvbiI6InRlc3QgdG9rZW4iLCJJbmhlcml0ZWRDbGFpbXMiOm51bGx9Fh11cf4WCED6vAIZjLr8bIcRy46XZFMknVCIYwVPOOKRtJtt8Jy8nLShT5kOgul6s_KN6MMcPyIYHlPcDsWKBw.bnVsbA

5. det token describe [-h] [--json | --yaml] token_id [token_id ...]

% det token describe 395 396 397
     ID |   User ID | Description   | Created At                  | Expires At                  | Revoked   | Token Type
------+-----------+---------------+-----------------------------+-----------------------------+-----------+--------------
  395 |         1 |               | 2024-10-11T17:42:32.079651Z | 2024-11-10T17:42:32.079651Z | False     | ACCESS_TOKEN
  396 |         2 | test token    | 2024-10-11T17:43:29.696547Z | 2024-10-12T23:43:29.696547Z | False     | ACCESS_TOKEN
  397 |         2 | test token    | 2024-10-11T17:43:38.265917Z | 2024-10-12T23:43:38.265917Z | False     | ACCESS_TOKEN

6. det token revoke [-h] token_id

% det token revoke 397           
{
  "tokenInfo": {
    "id": 397,
    "userId": 2,
    "createdAt": "2024-10-11T17:43:38.265917Z",
    "description": "test token",
    "expiry": "2024-10-12T23:43:38.265917Z",
    "revoked": true,
    "tokenType": "TOKEN_TYPE_ACCESS_TOKEN"
  }
}
Successfully updated token with ID: 397.

7. det token edit [-h] [--description DESCRIPTION] [--json | --yaml] token_id

% det token edit 396 -d "test token-1" 
{
  "tokenInfo": {
    "id": 396,
    "userId": 2,
    "createdAt": "2024-10-11T17:43:29.696547Z",
    "description": "test token-1",
    "expiry": "2024-10-12T23:43:29.696547Z",
    "revoked": false,
    "tokenType": "TOKEN_TYPE_ACCESS_TOKEN"
  }
}
Successfully updated token with ID: 396.

Test Plan

1. Login to latest-main Swagger as Admin and try all 3 APIs
2. Should notice permissions kick in when trying with non-Admin user
3. Try below CLI commands:
4. Try login and revoking the token and log in back

Completed local tests

• Table alter & Integration Tests of CRUD Ops
• API creations & Integration Tests
• CLI for CREATE, GET, PATCH API

After migration, you can see the altered 'user_sessions' table, along with CRUD operations to support POST, GET and PATCH operations.

Migration to create a table of the given schema:

$ cd master/static/migrations
$ ./migration-create.sh add-long-lived-tokens-table
$ cd ~/Projects/determined/
$ master/build/determined-master --config-file /Users/shreya/Projects/determined/.circleci/devcluster/master.yaml migrate set_version 20240829000038
$ master/build/determined-master --config-file /Users/shreya/Projects/determined/.circleci/devcluster/master.yaml migrate up 

To build and generate files

$ make -C proto build
$ make -C harness build
$ make -C bindings build
$ make -C master build

To mock

$ make -C master clean
$ make -C master check OR $ make -C master mocks

otherwise:

$ make clean
$ make all

To check:

$ make -C proto fmt
$ make -C master fmt
$ make -C proto check
$ make -C master check

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	e386613
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/6711912ef664c00007ba8fbb

[codecov]

Codecov Report

Attention: Patch coverage is 41.86047% with 250 lines in your changes missing coverage. Please review.

Project coverage is 54.37%. Comparing base (2ef2f12) to head (e386613).
Report is 44 commits behind head on main.

Files with missing lines	Patch %	Lines
harness/determined/cli/token.py	16.16%	83 Missing ⚠️
master/internal/token/authz_rbac.go	1.36%	72 Missing ⚠️
master/internal/token/authz_basic_impl.go	4.16%	23 Missing ⚠️
master/pkg/model/user.go	0.00%	22 Missing ⚠️
master/internal/api_token.go	80.95%	20 Missing ⚠️
master/internal/token/postgres_token.go	75.71%	17 Missing ⚠️
master/internal/token/authz_permissive.go	10.00%	9 Missing ⚠️
master/internal/api_auth.go	0.00%	2 Missing ⚠️
master/internal/user/postgres_users.go	92.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9867      +/-   ##
==========================================
- Coverage   54.42%   54.37%   -0.05%     
==========================================
  Files        1262     1268       +6     
  Lines      158880   159302     +422     
  Branches     3631     3630       -1     
==========================================
+ Hits        86463    86626     +163     
- Misses      72283    72542     +259     
  Partials      134      134              

Flag	Coverage Δ
backend	45.49% <49.54%> (+<0.01%)	⬆️
harness	72.55% <16.16%> (-0.20%)	⬇️
web	53.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
harness/determined/cli/cli.py	45.13% <ø> (ø)
master/internal/api_auth.go	41.26% <0.00%> (-0.34%)	⬇️
master/internal/user/postgres_users.go	81.10% <92.00%> (+0.59%)	⬆️
master/internal/token/authz_permissive.go	10.00% <10.00%> (ø)
master/internal/token/postgres_token.go	75.71% <75.71%> (ø)
master/internal/api_token.go	80.95% <80.95%> (ø)
master/pkg/model/user.go	23.30% <0.00%> (-4.62%)	⬇️
master/internal/token/authz_basic_impl.go	4.16% <4.16%> (ø)
master/internal/token/authz_rbac.go	1.36% <1.36%> (ø)
harness/determined/cli/token.py	16.16% <16.16%> (ø)

... and 5 files with indirect coverage changes

[azhou-determined]

nice work on this, it's come a long way!

[tara-hpe]

LGTM

[maxrussell]

I'm continuing to review, but this is my feedback so far.

[maxrussell]

Continuing to review, submitting some in-progress minor feedback