Build container image in two steps (new action) (#5)

* Drafting new action

* Adding a test for the action

* Missing image tag

* Adding docker buildx

* Fixing errors in workflow (replacing runner)

* Fixing args (registry)

* Second container file was not passed

* Adding write to the container registry

* Adding password to login to ghcr.io and fixing naming

* Using github actor as login

* Wrong location for dependencies tag in push 2

* Wrong location for dependencies tag in push 2 (again)

* Last bit

* Working on documentation

* Fixing path to container file

* Adding workflow dispatch to check if cache works

* Login should always happen

* Clarifying documentation and adding the entry to the README

* Adding a note on how caching works

* Update twostep-container-build/README.md

---------

Co-authored-by: Damon Bayer <[email protected]>

.github/workflows/test-post-artifact.yml

@@ -3,7 +3,6 @@ name: Test PR post artifact
 on:
   pull_request:
     branches: [main]
-  pull_request_target: 
     paths:
       - '.github/workflows/test-post-artifact.yml'
       - 'post-artifact/action.yml'

.github/workflows/test-twostep-container-build.yml

@@ -0,0 +1,38 @@
+name: Test twostep-container-build
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - '.github/workflows/test-twostep-container-build.yml'
+      - 'twostep-container-build/action.yml'
+  push:
+    branches: [main]
+    paths:
+      - '.github/workflows/test-twostep-container-build.yml'
+      - 'twostep-container-build/action.yml'
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: read
+      packages: write
+      pull-requests: write
+
+    steps:
+      - uses: actions/checkout@v4
+        name: Checkout code
+
+      - name: Two-step build
+        uses: ./twostep-container-build
+        with:
+          registry: ghcr.io/
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+          container-file-1: twostep-container-build/examples/Containerfile.dependencies
+          container-file-2: twostep-container-build/examples/Containerfile
+          first-step-cache-key: ${{ hashFiles('twostep-container-build/examples/Containerfile.dependencies') }}
+          image: cdcgov/cfa-actions

README.md

@@ -10,6 +10,7 @@
 This repo contains personalized actions designed by CDC's CFA team. Please use with caution as these actions are not officially supported by GitHub.
 
 - [post-artifact](./post-artifact): Post an artifact as a comment in a PR. Useful when you need to easily access a built element during a workflow such as a website, a report, etc.
+- [twostep-container-build](./twostep-container-build): Cache dependencies of a project by splitting the build process in two steps. The first step builds the dependencies and caches the image. The second step builds the project using the cached image as a base.
 
 
 ## Public Domain Standard Notice

twostep-container-build/README.md

@@ -0,0 +1,103 @@
+# Build image in two steps: Caching dependencies
+
+This action will build a container image for a project in two steps and push the image to a container registry. During the first step, using the container file `container-file-1`, it will build and cache the image containing the dependencies of the main project. After the first step, a second build and push process happens based on the container file `container-file-2`. The `container-file-2` uses as base image the one created during the first step.
+
+```mermaid
+flowchart LR
+  Containerfile1[container-file-1] -->|Generates|Image1
+  Image1-->|Is used as a baseline for|Containerfile2
+  Containerfile2-->|Generates|Image2
+```
+
+Caching is done using the [actions/cache](https://github.com/actions/cache/tree/v4) (lookup only) and [docker/build-push-action](https://github.com/docker/build-push-action) actions. Users have to explicitly provide the cache key for the first step. For example, if you are dealing with an R package, you can cache the dependencies by passing the key `${{ hashFiles('DESCRIPTION') }}` to the `first-step-cache-key` input. That way, the first step will only be executed if the dependencies change.
+
+## Inputs
+
+| Field | Description | Required | Default |
+|-------|-------------|----------|---------|
+| `container-file-1` | Path to the first container file | true | |
+| `container-file-2` | Path to the second container file | true | |
+| `first-step-cache-key` | Cache key for the first step | true | |
+| `image` | Name of the image | true | |
+| `username` | Username for the registry | true |  |
+| `password` | Password for the registry | true |  |
+| `registry` | Registry to push the image to | true |  |
+| `main-branch-name` | Name of the main branch | false | `'main'` |
+| `main-branch-tag` | Tag to use for the main branch | false | `'latest'` |
+| `push-image-1` | Push the image created during the first step | false | `false` |
+| `push-image-2` | Push the image created during the second step | false | `false` |
+
+## Example: Using ghcr.io
+
+The workflow is triggered on pull requests and pushes to the main branch. The image is pushed to `ghcr.io` and the image name is `cdcgov/cfa-actions` (full name is `ghcr.io/cdcgov/cfa-actions`). A functional version of this workflow is executed [here](../.github/workflows/test-twostep-container-build.yml).
+
+```yaml
+name: Building the container and put it on ghcr.io
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    # Since we are using ghcr.io, we need to set the permissions to write
+    # for the packages.
+    permissions:
+      contents: read
+      packages: write
+      pull-requests: write
+
+    steps:
+      - uses: actions/checkout@v4
+        name: Checkout code
+
+      - name: Two-step build
+        uses: ./[email protected]
+        with:
+          # Login information
+          registry: ghcr.io/
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+          # Paths to the container files
+          container-file-1: Containerfile.dependencies
+          container-file-2: Containerfile
+
+          # We are using the dependency container for caching
+          first-step-cache-key: ${{ hashFiles('Containerfile.dependencies') }}
+
+          # The image to build includes the organization (that's how it is
+          # on ghcr.io)
+          image: cdcgov/cfa-actions
+
+```
+
+The container files (which can be found under the [examples](examples) directory) have the following structure:
+
+[`Containerfile.dependencies`](examples/Containerfile.dependencies)
+
+```Containerfile
+FROM rocker/r-base:4.4.0
+
+RUN install2.r epiworldR
+
+CMD ["bash"]
+```
+
+[`Containerfile`](examples/Containerfile)
+
+```Containerfile
+ARG TAG=latest
+
+FROM ghcr.io/cdcgov/cfa-actions:${TAG}
+
+COPY twostep-container-build/example/Containerfile /app/.
+
+CMD ["bash"]
+```
+
+Notice the `TAG` argument which is passed to the second container file. During runs of the action, `TAG` takes the value of the branch name or `latest` if the branch is the main branch.

twostep-container-build/action.yml

@@ -0,0 +1,130 @@
+name: twostep-container-build
+description: |
+  
+inputs:
+  container-file-1:
+    description: |
+      The first container file to build.
+    required: true
+  container-file-2: 
+    description: |
+      The second container file to build.
+    required: true
+  first-step-cache-key:
+    description: |
+      The key used to cache the first step of the process. Usually
+      generated using the `hashFiles` function.
+  image:
+    description: |
+      The image to build. For instance, cfa-prod-batch, or cdcgov/cfa-actions.
+    required: true
+  username:
+    description: |
+      The username to use for the container registry login.
+    required: true
+  password:
+    description: |
+      The password to use for the container registry login.
+    required: true
+  registry:
+    description: |
+      The registry to use for the container registry login
+      with trailing slash. For example, ghcr.io/,
+      cfaprodbatchcr.azurecr.io/, etc.
+    required: true
+  main-branch-name:
+    description: |
+      The name of the repository's base branch. Defaults to main.
+    required: false
+    default: 'main'
+  main-branch-tag:
+    description: |
+      The tag to use for the main branch. For instance, latest.
+    required: false
+    default: 'latest'
+  push-image-1:
+    description: |
+      Whether to push the first image. For instance, true.
+    required: false
+    default: 'true'
+  push-image-2:
+    description: |
+      Whether to push the second image. For instance, true.
+    required: false
+    default: 'true'
+runs:
+  using: 'composite'
+
+  steps:
+
+    - name: Checkout code
+      uses: actions/checkout@v4
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+
+    - name: Getting the commit message
+      id: commit-message
+      run: echo "message=$(git log -1 --pretty=%s HEAD)" >> $GITHUB_OUTPUT
+      shell: bash
+
+    - name: Checking out the latest (may be merge if PR)
+      uses: actions/checkout@v4
+
+    # From: https://stackoverflow.com/a/58035262/2097171
+    - name: Extract branch name
+      shell: bash
+      run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
+      id: branch-name
+
+    #########################################################################
+    # Getting the tag
+    # The tag will be used for both the docker image and the batch pool
+    #########################################################################
+    - name: Figure out tag (either latest if it is main or the branch name)
+      shell: bash
+      id: image-tag
+      run: |
+        if [ "${{ steps.branch-name.outputs.branch }}" = "${{ inputs.main-branch-name }}" ]; then
+          echo "tag=${{ inputs.main-branch-tag }}" >> $GITHUB_OUTPUT
+        else
+          echo "tag=${{ steps.branch-name.outputs.branch }}" >> $GITHUB_OUTPUT
+        fi
+
+    - name: Check cache for base image
+      uses: actions/cache@v4
+      id: cache
+      with:
+        key: ${{ inputs.first-step-cache-key }}
+        lookup-only: true
+        path:
+          ${{ inputs.container-file-1 }}
+
+    - name: Login to the Container Registry
+      if: inputs.registry != ''
+      uses: docker/login-action@v3
+      with:
+        registry: ${{ inputs.registry }}
+        username: ${{ inputs.username }}
+        password: ${{ inputs.password }}
+
+    - name: Build and push
+      if: steps.cache.outputs.cache-hit != 'true'
+      uses: docker/build-push-action@v6
+      with:
+        no-cache: true
+        push: ${{ inputs.push-image-1 }}
+        tags: |
+          ${{ inputs.registry }}${{ inputs.image }}:dependencies-${{ steps.image-tag.outputs.tag }}
+        file: ${{ inputs.container-file-1 }}
+
+    - name: Build and push the main image
+      id: build_and_push_model_image
+      uses: docker/build-push-action@v6
+      with:
+        no-cache: true
+        push: ${{ inputs.push-image-2 }}
+        tags: |
+          ${{ inputs.registry }}${{ inputs.image }}:${{ steps.image-tag.outputs.tag }}
+        file: ${{ inputs.container-file-2 }}
+        build-args: |
+          TAG=dependencies-${{ steps.image-tag.outputs.tag }}

twostep-container-build/examples/Containerfile

@@ -0,0 +1,7 @@
+ARG TAG=latest
+
+FROM ghcr.io/cdcgov/cfa-actions:${TAG}
+
+COPY twostep-container-build/examples/Containerfile /app/.
+
+CMD ["bash"]

twostep-container-build/examples/Containerfile.dependencies

@@ -0,0 +1,5 @@
+FROM rocker/r-base:4.4.0
+
+RUN install2.r epiworldR
+
+CMD ["bash"]

twostep-container-build/examples/example.yaml

@@ -0,0 +1,38 @@
+name: Building the container and put it on ghcr.io
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    # Since we are using ghcr.io, we need to set the permissions to write
+    # for the packages.
+    permissions:
+      contents: read
+      packages: write
+      pull-requests: write
+
+    steps:
+      - uses: actions/checkout@v4
+        name: Checkout code
+
+      - name: Two-step build
+        uses: ./twostep-container-build
+        with:
+          # Login information
+          registry: ghcr.io/
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+          # Paths to the container files
+          container-file-1: Containerfile.dependencies
+          container-file-2: Containerfile
+          # We are using the dependency container for caching
+          first-step-cache-key: ${{ hashFiles('Containerfile.dependencies') }}
+          # The image to build includes the organization (that's how it is
+          # on ghcr.io)
+          image: cdcgov/cfa-actions