Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first draft of model registry kserve custom storage initializer #25

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ IMG_VERSION ?= main
# container image repository
IMG_REPO ?= model-registry
# container image
IMG := ${IMG_REGISTRY}/$(IMG_ORG)/$(IMG_REPO)
IMG ?= ${IMG_REGISTRY}/$(IMG_ORG)/$(IMG_REPO)

model-registry: build

Expand Down
19 changes: 19 additions & 0 deletions csi/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
bin/

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Go workspace file
go.work
32 changes: 32 additions & 0 deletions csi/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Build the model-registry binary
FROM registry.access.redhat.com/ubi8/go-toolset:1.19 as builder

WORKDIR /workspace
# Copy the Go Modules manifests
COPY ["go.mod", "go.sum", "./"]
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

USER root

# Copy the go source
COPY ["Makefile", "main.go", "./"]

# Copy rest of the source
COPY bin/ bin/
COPY pkg/ pkg/

# Build
USER root
RUN CGO_ENABLED=1 GOOS=linux GOARCH=amd64 make build

# Use distroless as minimal base image to package the model-registry storage initializer binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.8
WORKDIR /
# copy the storage initializer binary
COPY --from=builder /workspace/bin/mr-storage-initializer .
USER 65532:65532

ENTRYPOINT ["/mr-storage-initializer"]
249 changes: 249 additions & 0 deletions csi/GET_STARTED.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
# Get Started

Embark on your journey with this custom storage initializer by exploring a simple hello-world example. Learn how to seamlessly integrate and leverage the power of this tool in just a few steps.

## Prerequisites

* Install [Kind](https://kind.sigs.k8s.io/docs/user/quick-start) (Kubernetes in Docker) to run local Kubernetes cluster with Docker container nodes.
* Install the [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/), which allows you to run commands against Kubernetes clusters.
* Install the [Kustomize](https://kustomize.io/), which allows you to customize app configuration.

## Environment Preparation

We assume all [prerequisites](#prerequisites) are satisfied at this point.

### Create the environment

1. After having kind installed, create a kind cluster with:
```bash
kind create cluster
```

2. Configure `kubectl` to use kind context
```bash
kubectl config use-context kind-kind
```

3. Setup local deployment of *Kserve* using the provided *Kserve quick installation* script
```bash
curl -s "https://raw.githubusercontent.com/kserve/kserve/release-0.11/hack/quick_install.sh" | bash
```

4. Install *model registry* in the local cluster

[Optional ]Use model registry with local changes:

```bash
TAG=$(git rev-parse HEAD) && \
MR_IMG=quay.io/$USER/model-registry:$TAG && \
make -C ../ IMG_ORG=$USER IMG_VERSION=$TAG image/build && \
kind load docker-image $MR_IMG
```

then:

```bash
bash ./scripts/install_modelregistry.sh -i $MR_IMG
```

> _NOTE_: If you want to use a remote image you can simply remove the `-i` option

> _NOTE_: The `./scripts/install_modelregistry.sh` will make some change to [base/kustomization.yaml](../manifests/kustomize/base/kustomization.yaml) that you DON'T need to commit!!

5. [Optional] Use local container image for CSI

```bash
IMG=quay.io/$USER/model-registry-storage-initializer:$(git rev-parse HEAD) && make IMG=$IMG docker-build && kind load docker-image $IMG
```

## First InferenceService with ModelRegistry URI

In this tutorial, you will deploy an InferenceService with a predictor that will load a model indexed into the model registry, the indexed model refers to a scikit-learn model trained with the [iris](https://archive.ics.uci.edu/ml/datasets/iris) dataset. This dataset has three output class: Iris Setosa, Iris Versicolour, and Iris Virginica.

You will then send an inference request to your deployed model in order to get a prediction for the class of iris plant your request corresponds to.

Since your model is being deployed as an InferenceService, not a raw Kubernetes Service, you just need to provide the storage location of the model using the `model-registry://` URI format and it gets some super powers out of the box.


### Register a Model into ModelRegistry

Apply `Port Forward` to the model registry service in order to being able to interact with it from the outside of the cluster.
```bash
kubectl port-forward --namespace kubeflow svc/model-registry-service 8080:8080
```

And then (in another terminal):
```bash
export MR_HOSTNAME=localhost:8080
```

Then, in the same terminal where you exported `MR_HOSTNAME`, perform the following actions:
1. Register an empty `RegisteredModel`

```bash
curl --silent -X 'POST' \
"$MR_HOSTNAME/api/model_registry/v1alpha2/registered_models" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"description": "Iris scikit-learn model",
"name": "iris"
}'
```

Expected output:
```bash
{"createTimeSinceEpoch":"1709287882361","customProperties":{},"description":"Iris scikit-learn model","id":"1","lastUpdateTimeSinceEpoch":"1709287882361","name":"iris"}
```

2. Register the first `ModelVersion`

```bash
curl --silent -X 'POST' \
"$MR_HOSTNAME/api/model_registry/v1alpha2/model_versions" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"description": "Iris model version v1",
"name": "v1",
"registeredModelID": "1"
}'
```

Expected output:
```bash
{"createTimeSinceEpoch":"1709287890365","customProperties":{},"description":"Iris model version v1","id":"2","lastUpdateTimeSinceEpoch":"1709287890365","name":"v1"}
```

3. Register the raw `ModelArtifact`

This artifact defines where the actual trained model is stored, i.e., `gs://kfserving-examples/models/sklearn/1.0/model`

```bash
curl --silent -X 'POST' \
"$MR_HOSTNAME/api/model_registry/v1alpha2/model_versions/2/artifacts" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"description": "Model artifact for Iris v1",
"uri": "gs://kfserving-examples/models/sklearn/1.0/model",
"state": "UNKNOWN",
"name": "iris-model-v1",
"modelFormatName": "sklearn",
"modelFormatVersion": "1",
"artifactType": "model-artifact"
}'
```

Expected output:
```bash
{"artifactType":"model-artifact","createTimeSinceEpoch":"1709287972637","customProperties":{},"description":"Model artifact for Iris v1","id":"1","lastUpdateTimeSinceEpoch":"1709287972637","modelFormatName":"sklearn","modelFormatVersion":"1","name":"iris-model-v1","state":"UNKNOWN","uri":"gs://kfserving-examples/models/sklearn/1.0/model"}
```

> _NOTE_: double check the provided IDs are the expected ones.

### Apply the `ClusterStorageContainer` resource

Retrieve the model registry service and MLMD port:
```bash
MODEL_REGISTRY_SERVICE=model-registry-service
MODEL_REGISTRY_REST_PORT=$(kubectl get svc/$MODEL_REGISTRY_SERVICE -n kubeflow --output jsonpath='{.spec.ports[0].targetPort}' )
```

Apply the cluster-scoped `ClusterStorageContainer` CR to setup configure the `model registry storage initilizer` for `model-registry://` URI formats.

```bash
kubectl apply -f - <<EOF
apiVersion: "serving.kserve.io/v1alpha1"
kind: ClusterStorageContainer
metadata:
name: mr-initializer
spec:
container:
name: storage-initializer
image: $IMG
env:
- name: MODEL_REGISTRY_BASE_URL
value: "$MODEL_REGISTRY_SERVICE.kubeflow.svc.cluster.local:$MODEL_REGISTRY_REST_PORT"
- name: MODEL_REGISTRY_SCHEME
value: "http"
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 1Gi
cpu: "1"
supportedUriFormats:
- prefix: model-registry://

EOF
```

> _NOTE_: as `$IMG` you could use either the one created during [env preparation](#environment-preparation) or any other remote img in the container registry.

### Create an `InferenceService`

1. Create a namespace
```bash
kubectl create namespace kserve-test
```

2. Create the `InferenceService`
```bash
kubectl apply -n kserve-test -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "iris-model"
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: "model-registry://iris/v1"
EOF
```

3. Check `InferenceService` status
```bash
kubectl get inferenceservices iris-model -n kserve-test
```

4. Determine the ingress IP and ports

```bash
kubectl get svc istio-ingressgateway -n istio-system
```

And then:
```bash
INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')
kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8081:80
```

After that (in another terminal):
```bash
export INGRESS_HOST=localhost
export INGRESS_PORT=8081
```

5. Perform the inference request

Prepare the input data:
```bash
cat <<EOF > "/tmp/iris-input.json"
{
"instances": [
[6.8, 2.8, 4.8, 1.4],
[6.0, 3.4, 4.5, 1.6]
]
}
EOF
```

If you do not have DNS, you can still curl with the ingress gateway external IP using the HOST Header.
```bash
SERVICE_HOSTNAME=$(kubectl get inferenceservice iris-model -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" "http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/iris-v1:predict" -d @/tmp/iris-input.json
```
37 changes: 37 additions & 0 deletions csi/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
IMG ?= quay.io/${USER}/model-registry-storage-initializer:latest

.PHONY: help
help: ## Display this help.
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

##@ Development

.PHONY: fmt
fmt: ## Run go fmt against code.
go fmt ./...

.PHONY: vet
vet: ## Run go vet against code.
go vet ./...

.PHONY: test
test: fmt vet ## Run tests.
go test ./... -coverprofile cover.out

##@ Build

.PHONY: build
build: fmt vet ## Build binary.
go build -o bin/mr-storage-initializer main.go

.PHONY: run
run: fmt vet ## Run the program
go run ./main.go $(SOURCE_URI) $(DEST_PATH)

.PHONY: docker-build
docker-build: test ## Build docker image.
docker build . -f ./Dockerfile -t ${IMG}

.PHONY: docker-push
docker-push: ## Push docker image.
docker push ${IMG}
Loading
Loading