Skip to content

Latest commit

 

History

History
161 lines (109 loc) · 7.46 KB

DevOps.md

File metadata and controls

161 lines (109 loc) · 7.46 KB
Raw

Project Deliverable #4: Implementation of User Stories Set #1

Team: Devops 345

The following document contains the methodology we'll adopt to support the Backend teams, along with the experiments run and the POCs done.

Requirements Gathering and Resource Acquisition

We collaborated with Backend teams 3, 4, 5 to better understand their requirements and tools and technologies they would use to develop their respective backends. The following is a consolidated list of the same:

Team Tech Stack
Backend 3 Java, SpringBoot, Gradle, PostgresSQL, RabbitMQ
Backend 4 Java, SpringBoot, Maven, RabbitMQ, Redis (optional)
Backend 5 Java, SpringBoot, SQL/NoSQL DB, MQ (Kafka)

After discussion with TAs and the professor, we have finalized Microsoft Azure as our cloud provider and have been provisioned $200 ($50 credits per person) worth of Azure credits.

Design

As per the best practices, we'd like to accomplish this in two parts: CI and CD.

The CI part refers to the pipeline which will comprise the following steps:

  1. Compile, test and build the code in a docker container.
  2. Create a dockerized image for the artifact generated in the previous step.
  3. Push it to a central repository, such as Docker Hub, Azure Container Registry, etc.

The CD part refers to the pipeline which will comprise the following steps:

  1. Create a containerized instance of required DB and deploy it on cloud.
  2. Pull the docker image from the Docker registry (pushed in the step 3 of the CI pipeline) and deploy it on Azure Container Instances.

Since the teams were not yet ready with their implementations, in order to test our workflow, we created two pipelines. 1st was for Java + SpringBoot + Maven and 2nd was for Java + SpringBoot + Gradle, because those are the primary tech stacks of our backend teams.

The link to repos are as follows (They are private for now):

Repo 1 (Maven)- https://github.com/shaktimann/TestAddressBook

Repo 2 (Gradle)- https://github.com/aichkay01/TSFSBackendGradle

CI pipeline - Implementation

Following are the steps to create a CI pipeline for a backend team.

  1. Create a multi-stage Dockerfile to build in the 1st stage and dockerize and push docker image in the 2nd stage.

A sample Dockerfile (works for the sample project) looks like follows:

FROM maven:3.8.7 as BUILD_IMAGE
ENV APP_HOME=/root/dev/myapp/
RUN mkdir -p $APP_HOME/src/main/java
WORKDIR $APP_HOME
COPY . .
RUN mvn clean -B package -e -X --file pom.xml

FROM openjdk:17-jdk-alpine
WORKDIR /root/
COPY --from=BUILD_IMAGE /root/dev/myapp/target/AddressBook-1.0-SNAPSHOT.jar .
CMD java -jar AddressBook-1.0-SNAPSHOT.jar

  1. Create a Standard D2sv3 VM in Azure and set it up as a self hosted runner to execute builds.

    i. Create a VM with the help of: https://learn.microsoft.com/en-us/azure/virtual-machines/windows/quick-create-portal

    ii. Go to your git repo > Settings > Actions > Runners > New self-hosted runner

    iii. Choose appropriate settings. Login to your VM and follow the enlisted steps to setup your VM as a runner.

  1. Go to your git repo > Settings > Secrets and variables > Actions. Add Secrets for DOCKER_USERNAME and DOCKER_PASSWORD.

  2. Go to your git repo > Actions > New workflow.

  3. Create a new workflow to build your SpringBoot project, build a docker image from the packaged jar and push the docker image to Docker Hub / Azure Container Reqgistry.

A sample main.yaml (works for the sample project) looks like follows (Note that the value for 'runs-on' needs to be configured as per the label of your runner, obtained after step 2.):

# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Java CI with Maven

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
        
  build_and_push:
    runs-on: self-hosted
    steps:
    - uses: actions/checkout@v3
    - name: Docker build
      run: |
        docker build . -t manunigotiya/address-book:spring-docker
    - name: Login to DockerHub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKER_PASSWORD }}

    - name: Docker push
      run: |
        docker push manunigotiya/address-book:spring-docker
  1. Commit the main.yaml file.

CD pipeline - Workflow

For continuous delivery, we have broken down the workflow into two components:

  • Deployment of the application.
  • Deployment of a DB instance as per team's requirements.

To prepare for future implementation regarding Continuous Delivery, we identified some avenues for exploration. As we did not have access to the class Azure resources, we couldn't develop a POC.

With regards to deploying containers, we have found three relevant ways:

  1. Docker-Azure Integration. Allows us to use Docker CLI or VSCode extensions to deploy containers with native Docker commands to Azure Container Instances (ACI) https://docs.docker.com/cloud/aci-integration/

  2. Azure CLI and Azure Container Registry. Native to Azure, the registry stores container images and the CLI is used to deploy them https://learn.microsoft.com/en-us/azure/container-instances/container-instances-tutorial-prepare-app

  3. GitHub Actions. A pipeline can be created to upload Docker images to Azure Container Registry and then deploy these containers directly on Azure.

    https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=userlevel

    https://docs.github.com/en/actions/deployment/deploying-to-your-cloud-provider/deploying-to-azure/deploying-docker-to-azure-app-service

We are aiming to focus on option 3 as it will further extend our CI pipeline as it is now.

We also looked into deploying databases on Azure and aim to begin with PostgreSQL databases. Below are some relevant links

As discussed in the class, we'll also evaluate and use the steps mentioned here to deploy Azure multi-container apps: https://github.com/alfredodeza/azure-multi-container-apps

Teams specifics and Strategy

Backend Team 3 - The team proposes to use RabbitMQ to build a notification service. As they progress and make a final decision on this, we'll begin to look into providing them with MQ services.

Backend Team 4 - The team proposes to use RabbitMQ to build a notification service. They have also mentioned using Redis as optional for caching. As they progress and make a final decision on this, we'll provide them with required services.

Backend Team 5 - The team is yet to choose between SQL or NoSQL DB. We'll offer them support accordingly.