Skip to content

Commit

Permalink
Update send data scripts, installation instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
RobGeada committed May 16, 2024
1 parent 027b7a8 commit 4493c48
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 14 deletions.
13 changes: 9 additions & 4 deletions 1-Installation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,20 +36,25 @@ your cluster management UI (for example, on console.redhat.com)
## ODH v2.x
If the provided ODH version in your cluster's OperatorHub is version 2.x, use the following steps:

### Prerequisties
1) Install the community Authorino Operator. At the time of writing, Authorino 0.11.1 is verified to be compatible.

### Install ODH (ODH v2.x)
1) Navigate to your `opendatahub` project
2) From "Installed Operators", select "Open Data Hub Operator".
3) Navigate to the "Data Science Cluster" tab and hit "Create DataScienceCluster"
4) In the YAML view Make sure `trustyai` is set to `Managed`:
3) Navigate to the "DSC Initialization" tab and hit "Create DSCInitialization", then install the default DSCI.
4) Navigate to the "Data Science Cluster" tab and hit "Create DataScienceCluster"
5) In the YAML view Make sure `trustyai` is set to `Managed`:
![ODH V2 YAML](images/odh_V2.png)
5) Hit the "Create" button
6) Within the "Pods" menu, you should begin to see various ODH components being created, including the `trustyai-service-operator-controller-manager-xxx`
6) Hit the "Create" button
7) Within the "Pods" menu, you should begin to see various ODH components being created, including the `trustyai-service-operator-controller-manager-xxx`

### Install TrustyAI (ODH v2.x)
1) Navigate to your `model-namespace` project: `oc project model-namespace`
2) Run `oc apply -f resources/trustyai_crd.yaml`. This will install the TrustyAI Service
into your `model-namespace` project, which will then provide TrustyAI features to all subsequent models deployed into that project, such as explainability, fairness monitoring, and data drift monitoring,

# Legacy Installation Steps
## ODH v1.x (legacy)
If the provided ODH version in your cluster's OperatorHub is version 1.x, use the following steps:
### Install ODH (ODH v1.x)
Expand Down
4 changes: 2 additions & 2 deletions 2-BiasMonitoring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ use the following information about the applicant to make their prediction:
* Length of Employment (in days)

What we want to verify is that neither of our models are not biased over the gender field of `Is Male-Identifying?`. To do this,
we will monitor our models with *Statistical Parity Difference (SPD)* metric, which will tell us how the difference betweem how often
we will monitor our models with *Statistical Parity Difference (SPD)* metric, which will tell us how the difference between how often
male-identifying and non-male-identifying applicants are given favorable predictions (i.e., they are predicted
to pay back their loans). Ideally, the SPD value would be 0, indicating that both groups have equal likelihood of getting a good outcome. However, an SPD value between -0.1 and 0.1 is also indicative of reasonable fairness,
indicating that the two groups' rates of getting good outcomes only varies by +/-10%.
Expand All @@ -38,7 +38,7 @@ an instance of the TrustyAI Service.

> ✏️ TrustyAI endpoints are authenticated via a Bearer token. To obtain this token, run the following command:
> ```shell
>export TOKEN=$(oc whoami -t)
> export TOKEN=$(oc whoami -t)
> ```
## Deploy Models
Expand Down
17 changes: 9 additions & 8 deletions 2-BiasMonitoring/scripts/send_data_batch
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,21 @@ MODEL_ALPHA=demo-loan-nn-onnx-alpha
MODEL_BETA=demo-loan-nn-onnx-beta

ODHPROJECT=opendatahub
MM_NAMESPACE=model-namespace
MM_NAMESPACE=opendatahub-model

oc project $MM_NAMESPACE 2>&1 1>/dev/null
TRUSTY_POD=$(oc get pods -o name | grep trustyai-service)
TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

INFER_ROUTE_ALPHA=$(oc get route demo-loan-nn-onnx-alpha --template={{.spec.host}}{{.spec.path}})
INFER_ROUTE_BETA=$(oc get route demo-loan-nn-onnx-beta --template={{.spec.host}}{{.spec.path}})

function curl_trustyai_token() {
TOKEN=$(oc create token user-one -n ${MM_NAMESPACE}) || eval "$FAILURE_HANDLING"
TOKEN=$(oc whoami -t)

function curl_token() {
curl -s -H "Authorization: Bearer ${TOKEN}" "$@"
}


function check_for_reception () {
CHECKS=0
THRESHOLD=${3:-950}
Expand All @@ -47,11 +48,11 @@ function check_for_reception () {
}

function get_counter () {
FILE_PRESENT=$(curl_trustyai_token $TRUSTY_ROUTE/info | grep $1 || echo '')
FILE_PRESENT=$(curl_token $TRUSTY_ROUTE/info | grep $1 || echo '')
if [[ -z $FILE_PRESENT ]]; then
echo 0
else
echo $(curl_trustyai_token $TRUSTY_ROUTE/info | jq ".[].data | select(.modelId==\"$1\")" | jq .observations)
echo $(curl_token $TRUSTY_ROUTE/info | jq ".[].data | select(.modelId==\"$1\")" | jq .observations)
fi
}

Expand All @@ -76,12 +77,12 @@ while [ "$SUCCESS_ALPHA" != true ] || [ "$SUCCESS_BETA" != true ]; do
fi

if [ "$SUCCESS_ALPHA" != true ]; then
curl_trustyai_token https://$INFER_ROUTE_ALPHA/infer -d @$1 > /dev/null 2>&1
curl_token https://$INFER_ROUTE_ALPHA/infer -d @$1 > /dev/null 2>&1
( check_for_reception $(( $(jq '.inputs[0].data | length' $1) + $START_OBS_ALPHA )) $MODEL_ALPHA; ) && SUCCESS_ALPHA=true
fi;

if [ "$SUCCESS_BETA" != true ]; then
curl_trustyai_token https://$INFER_ROUTE_BETA/infer -d @$1 > /dev/null 2>&1
curl_token https://$INFER_ROUTE_BETA/infer -d @$1 > /dev/null 2>&1
( check_for_reception $(( $(jq '.inputs[0].data | length' $1) + $START_OBS_BETA )) $MODEL_BETA; ) && SUCCESS_BETA=true
fi;

Expand Down

0 comments on commit 4493c48

Please sign in to comment.