MLFLOW_TRACKING_EXTERNAL_URI is empty. It seems MLFlow Tracking Server is not properly deployed

[davidsinjaya]

Hi,

I have an endless loop when installing mlops-composer-mlflow.
as far as I understand the code, there is no specific script to install MLFlow Tracking Server.

Problematic paragraph

mlops-on-gcp/environments_setup/mlops-composer-mlflow/install.sh

Line 217 in 54b9a77

MLFLOW_TRACKING_EXTERNAL_URI="https://"

MLFLOW_TRACKING_EXTERNAL_URI="https://"
# Internal access from Composer to Mlflow
MLFLOW_URI_FOR_COMPOSER=="http://"
while [ "$MLFLOW_TRACKING_EXTERNAL_URI" == "https://" ] || [ "$MLFLOW_URI_FOR_COMPOSER" == "http://" ]
do
  echo "wait 5 seconds..."
  sleep 5s
  MLFLOW_TRACKING_EXTERNAL_URI="https://"$(kubectl describe configmap inverse-proxy-config -n mlflow | grep "googleusercontent.com")
  MLFLOW_URI_FOR_COMPOSER="http://"$(kubectl get svc -n mlflow mlflow -o jsonpath='{.spec.clusterIP}{":"}{.spec.ports[0].port}')
done

Command line output:

Waiting for MLflow Tracking server provisioned
wait 5 seconds...
Error on line: 234
Caused by: MLFLOW_TRACKING_EXTERNAL_URI="https://"$(kubectl describe configmap inverse-proxy-config -n mlflow | grep "googleusercontent.com")
That returned exit status: 1
Aborting...
wait 5 seconds...
Error on line: 234
Caused by: MLFLOW_TRACKING_EXTERNAL_URI="https://"$(kubectl describe configmap inverse-proxy-config -n mlflow | grep "googleusercontent.com")
That returned exit status: 1
Aborting...
^C

Debug code:

_<username>_@cloudshell:~/mlops-composer-mlflow _(<gcp_project_name>)_$ kubectl describe configmap inverse-proxy-config -n mlflow
Name:         inverse-proxy-config
Namespace:    mlflow
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=mlflow
Annotations:  meta.helm.sh/release-name: mlflow
              meta.helm.sh/release-namespace: mlflow

Data
====

BinaryData
====

Events:  <none>

1. I have managed to install the docker images for MLFlow docker image and MLFlow UI proxy image. Everything up to that problematic line has worked. Please let me know how to fix the problem

[kornelregius]

Hi davidsinjaya,
What is the output when you run
kubectl describe configmap inverse-proxy-config -n mlflow | grep "googleusercontent.com"
?

MLFlow Tracking Server should have been installed in kubernetes cluster by helm chart.

[davidsinjaya]

hi @kornelregius, thank you for the reply!
kubectl describe configmap inverse-proxy-config -n mlflow | grep "googleusercontent.com"
above command return nothing.

in original issue description i run the command kubectl describe configmap inverse-proxy-config -n mlflow, you can see the full content.

somehow, the code endless loop since "googleusercontent.com" is never written to the file

[kornelregius]

We should see the logs in proxy container. This container runs mlflow-helm/proxy/attempt-register-vm-on-proxy.sh to create a reverse proxy. Try to attach to this instance and run /opt/proxy/attempt-register-vm-on-proxy.sh manually.

[davidsinjaya]

I managed to do the following:

1. Login into mlflow container and run shell commands
2. Copy the mlflow-helm folder to the container (because I check the opt folder is empty in the container, I assumed this is what I should do)
3. Run /opt/proxy/attempt-register-vm-on-proxy.sh manually. using sh command

I sent the command line output error to your email. hoping that we can communicate through email instead hence I can send the screenshots. really appreciate your help!

[alexlgeorgiou]

Also having this issue - did you find a resolution?

[davidsinjaya]

hi @alexlgeorgiou , good to know that someone else has the same issue.
I am still waiting for @kornelregius 's kind response on this. 🙏
will let you know once we found how to fix this

[davidsinjaya]

UPDATE: i have not received an update yet from the author.

[jaganlal]

Any updates regarding the same?