Merge pull request #500 from GoogleCloudPlatform/cicd_build_update

Update KFP cicd for a cloud build service account change
GoogleCloudPlatform · Aug 6, 2024 · efc0759 · efc0759
2 parents 6c8cb90 + 99ac6f8
commit efc0759
Show file tree

Hide file tree

Showing 5 changed files with 105 additions and 9 deletions.
diff --git a/notebooks/kubeflow_pipelines/cicd/labs/cloudbuild_vertex.yaml b/notebooks/kubeflow_pipelines/cicd/labs/cloudbuild_vertex.yaml
@@ -47,6 +47,8 @@ steps:
   - |
     python $_PIPELINE_FOLDER/kfp-cli_vertex/run_pipeline.py  # TODO
 
+logsBucket: 'gs://$PROJECT_ID-cicd-log'
+
 # Push the images to Artifact Registry
 # TODO: List the images to be pushed to the project Docker registry
 images: # TODO

diff --git a/notebooks/kubeflow_pipelines/cicd/labs/kfp_cicd_vertex.ipynb b/notebooks/kubeflow_pipelines/cicd/labs/kfp_cicd_vertex.ipynb
@@ -40,7 +40,8 @@
     "PROJECT_ID = !(gcloud config get-value project)\n",
     "PROJECT_ID = PROJECT_ID[0]\n",
     "REGION = \"us-central1\"\n",
-    "ARTIFACT_STORE = f\"gs://{PROJECT_ID}-kfp-artifact-store\""
+    "ARTIFACT_STORE = f\"gs://{PROJECT_ID}-kfp-artifact-store\"\n",
+    "os.environ[\"REGION\"] = REGION"
    ]
   },
   {
@@ -202,6 +203,8 @@
     "  - |\n",
     "    python $_PIPELINE_FOLDER/kfp-cli_vertex/run_pipeline.py  # TODO\n",
     "\n",
+    "logsBucket: 'gs://$PROJECT_ID-cloudbuild'\n",
+    "\n",
     "# Push the images to Artifact Registry\n",
     "# TODO: List the images to be pushed to the project Docker registry\n",
     "images: # TODO\n",
@@ -211,6 +214,42 @@
     "timeout: 10800s\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a GCS bucket to save the build log."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "BUCKET = PROJECT_ID + \"-cicd-log\"\n",
+    "os.environ[\"BUCKET\"] = BUCKET"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "\n",
+    "exists=$(gsutil ls -d | grep -w gs://${BUCKET}/)\n",
+    "if [ -n \"$exists\" ]; then\n",
+    "    echo -e \"Bucket exists, let's not recreate it.\"\n",
+    "else\n",
+    "    echo \"Creating a new GCS bucket.\"\n",
+    "    gsutil mb -l ${REGION} gs://${BUCKET}\n",
+    "    echo \"Here are your current buckets:\"\n",
+    "    gsutil ls\n",
+    "fi"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -310,6 +349,7 @@
     "|Tag (regex)|.\\*|\n",
     "|Build Configuration|Cloud Build configuration file (yaml or json)|\n",
     "|Cloud Build configuration file location| ./notebooks/kubeflow_pipelines/cicd/labs/cloudbuild_vertex.yaml|\n",
+    "|Service account| `<PROJECT NUMBER>[email protected]` |\n",
     "\n",
     "\n",
     "Use the following values for the substitution variables:\n",

diff --git a/notebooks/kubeflow_pipelines/cicd/solutions/cloudbuild_vertex.yaml b/notebooks/kubeflow_pipelines/cicd/solutions/cloudbuild_vertex.yaml
@@ -51,6 +51,8 @@ steps:
   - |
     python $_PIPELINE_FOLDER/kfp-cli_vertex/run_pipeline.py --project_id=$PROJECT_ID --template_path=$_PIPELINE_FOLDER/pipeline_vertex/covertype_kfp_pipeline.yaml --display_name=coverype_kfp_pipeline --region=$_REGION
 
+logsBucket: 'gs://$PROJECT_ID-cicd-log'
+
 # Push the images to Artifact Registry
 images: ['us-docker.pkg.dev/$PROJECT_ID/asl-artifact-repo/trainer_image_covertype_vertex:latest']
 

diff --git a/notebooks/kubeflow_pipelines/cicd/solutions/kfp_cicd_vertex.ipynb b/notebooks/kubeflow_pipelines/cicd/solutions/kfp_cicd_vertex.ipynb
@@ -24,6 +24,17 @@
     "In this lab you will walk through authoring of a **Cloud Build** CI/CD workflow that automatically builds, deploys, and runs a Kubeflow pipeline on Vertex AI. You will also integrate your workflow with **GitHub** by setting up a trigger that starts the  workflow when a new tag is applied to the **GitHub** repo hosting the pipeline's code."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -34,13 +45,16 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "PROJECT_ID = !(gcloud config get-value project)\n",
     "PROJECT_ID = PROJECT_ID[0]\n",
     "REGION = \"us-central1\"\n",
-    "ARTIFACT_STORE = f\"gs://{PROJECT_ID}-kfp-artifact-store\""
+    "ARTIFACT_STORE = f\"gs://{PROJECT_ID}-kfp-artifact-store\"\n",
+    "os.environ[\"REGION\"] = REGION"
    ]
   },
   {
@@ -98,7 +112,9 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!gcloud builds submit --timeout 15m --tag {KFP_CLI_IMAGE_URI} kfp-cli_vertex"
@@ -122,6 +138,46 @@
     "The **Cloud Build** workflow configuration uses both standard and custom [Cloud Build builders](https://cloud.google.com/cloud-build/docs/cloud-builders). The custom builder encapsulates **KFP CLI**. "
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a GCS bucket to save the build log."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "BUCKET = PROJECT_ID + \"-cicd-log\"\n",
+    "os.environ[\"BUCKET\"] = BUCKET"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "\n",
+    "exists=$(gsutil ls -d | grep -w gs://${BUCKET}/)\n",
+    "if [ -n \"$exists\" ]; then\n",
+    "    echo -e \"Bucket exists, let's not recreate it.\"\n",
+    "else\n",
+    "    echo \"Creating a new GCS bucket.\"\n",
+    "    gsutil mb -l ${REGION} gs://${BUCKET}\n",
+    "    echo \"Here are your current buckets:\"\n",
+    "    gsutil ls\n",
+    "fi"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -204,6 +260,7 @@
     "|Tag (regex)|.\\*|\n",
     "|Build Configuration|Cloud Build configuration file (yaml or json)|\n",
     "|Cloud Build configuration file location| ./notebooks/kubeflow_pipelines/cicd/solutions/cloudbuild_vertex.yaml|\n",
+    "|Service account| `<PROJECT NUMBER>[email protected]` |\n",
     "\n",
     "\n",
     "Use the following values for the substitution variables:\n",

diff --git a/scripts/setup_on_cloudshell.sh b/scripts/setup_on_cloudshell.sh
@@ -17,11 +17,6 @@
 PROJECT_ID=$(gcloud config list project --format "value(core.project)")
 PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
 
-# Grant Editor role to Cloud Build service account
-gcloud projects add-iam-policy-binding $PROJECT_ID \
-  --member serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
-  --role roles/editor
-
 # Grant Storage Object Admin to Compute Engine service account
 gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member serviceAccount:$PROJECT_NUMBER[email protected] \