Cloud Migration Workflow Management

This is a subsystem of the Cloud-Barista platform that provides workflow management for cloud migration.

Overview

Create and management workflow through Airflow.
Create workflow based on gusty.

Development environment

Tested operating systems (OSs):
- Ubuntu 24.04, Ubuntu 22.04, Ubuntu 18.04
Language:
- Go: 1.23.0

How to run

1. Build and run

Build and run the binary with Airflow server

make run

Or, you can run it within Docker by this command.

make run_docker

If you want to stop the binary with Airflow server, run this command.

make stop

2. Write configuration file

Configuration file name is 'cm-cicada.yaml'
The configuration file must be placed in one of the following directories.
- .cm-cicada/conf directory under user's home directory
- 'conf' directory where running the binary
- 'conf' directory where placed in the path of 'CMCICADA_ROOT' environment variable
Configuration options
- task_component
  - load_examples : Load task component examples if true.
  - examples_directory : Specify directory where task component examples are located. Must be set if 'load_examples' is true.
- workflow_template
  - templates_directory : Specify directory where workflow templates are located.
- airflow-server
  - address : Specify Airflow server's address ({IP or Domain}:{Port})
  - use_tls : Must be true if Airflow server uses HTTPS.
  - skip_tls_verify : Skip TLS/SSL certificate verification. Must be set if 'use_tls' is true.
  - init_retry : Retry count of initializing Airflow server connection used by cm-cicada.
  - timeout : HTTP timeout value as seconds.
  - username : Airflow login username.
  - password : Airflow login password.
  - connections : Pre-define Airflow connections (Set multiple connections)
    - id : ID of connection
    - type : Type of connection
    - description : Description of connection
    - host : Host address or URL of connection
    - port : Port number for use connection
    - schema : Connection schema
    - login : Username for use connection
    - password : Password for use connection
- dag_directory_host : Specify DAG directory of the host. (Mounted DAG directory used by Airflow container.)
- dag_directory_container : Specify DAG directory of Airflow container. (DAG directory inside the container.)
listen
- port : Listen port of the API.

Configuration file example

cm-cicada:
  task_component:
      load_examples: true
      examples_directory: "./lib/airflow/example/task_component/"
  workflow_template:
      templates_directory: "./lib/airflow/example/workflow_template/"
  airflow-server:
      address: 127.0.0.1:8080
      use_tls: false
      # skip_tls_verify: true
      init_retry: 5
      timeout: 10
      username: "airflow"
      password: "airflow_pass"
      connections:
        - id: honeybee_api
          type: http
          description: HoneyBee API
          host: 127.0.0.1
          port: 8081
          schema: http
        - id: beetle_api
          type: http
          description: Beetle API
          host: 127.0.0.1
          port: 8056
          schema: http
          login: default
          password: default
        - id: tumblebug_api
          type: http
          description: TumbleBug API
          host: 127.0.0.1
          port: 1323
          schema: http
          login: default
          password: default
  dag_directory_host: "./_airflow/airflow-home/dags"
  dag_directory_container: "/usr/local/airflow/dags" # Use dag_directory_host for dag_directory_container, if this value is empty
  listen:
      port: 8083

3. Check workflow template

Check workflow template list.

curl -X 'GET' \
  'http://127.0.0.1:8083/cicada/workflow_template' \
  -H 'accept: application/json'

Get workflow template and copy the content.

curl -X 'GET' \
  'http://127.0.0.1:8083/cicada/workflow_template/81bbeb23-2c48-4536-9f01-55796e0fa394' \
  -H 'accept: application/json'

4. Create the workflow

Create the workflow by pasting copied workflow template content. Modify all of 'sgId' params. (Create the source group from honeybee.)

Check the ID of the created workflow.

curl -X 'POST' \
  'http://127.0.0.1:8083/cicada/workflow' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "migrate_infra_workflow",
  "data": {
    "description": "Migrate Server",
    "task_groups": [
      {
        "name": "migrate_infra",
        "description": "Migrate Server",
        "tasks": [
          {
            "name": "infra_import",
            "task_component": "honeybee_task_import_infra",
            "request_body": "",
            "path_params": {
              "sgId": "ba695bbb-d673-4092-9821-c9cb05676228"
            },
            "dependencies": []
          },
          {
            "name": "infra_get",
            "task_component": "honeybee_task_get_infra_refined",
            "request_body": "",
            "path_params": {
              "sgId": "ba695bbb-d673-4092-9821-c9cb05676228"
            },
            "dependencies": [
              "infra_import"
            ]
          },
          {
            "name": "infra_recommend",
            "task_component": "beetle_task_recommend_infra",
            "request_body": "infra_get",
            "path_params": null,
            "dependencies": [
              "infra_get"
            ]
          },
          {
            "name": "infra_migration",
            "task_component": "beetle_task_infra_migration",
            "request_body": "infra_recommend",
            "path_params": null,
            "dependencies": [
              "infra_recommend"
            ]
          },
          {
            "name": "register_target_to_source_group",
            "task_component": "honeybee_register_target_info_to_source_group",
            "request_body": "infra_migration",
            "path_params": {
              "sgId": "ba695bbb-d673-4092-9821-c9cb05676228"
            },
            "dependencies": [
              "infra_migration"
            ]
          }
        ]
      }
    ]
  }
}'

5. Run the workflow

curl -X 'POST' \
  'http://127.0.0.1:8083/cicada/workflow/4420da6c-c50f-4d8b-bc2e-f02d8b557fad/run' \
  -H 'accept: application/json' \
  -d ''

About Task Component

Each task in the workflow references a Task Component.

The Task Component part frequently leads to mistakes, and when the Request Body structure is complex or changes, it becomes difficult to keep up. Therefore, we have implemented a feature that automatically generates Task Components by reading JSON files.

As shown below, using JSONs containing name, description, api_connection_id, swagger_yaml_endpoint, and endpoint, the cicada reads the Swagger YAML and finds the endpoint to automatically construct task components.

The api_connection_id corresponds to one of the connection ids defined in https://github.com/cloud-barista/cm-cicada/blob/main/conf/cm-cicada.yaml

{
  "name": "tumblebug_mci_dynamic",
  "description": "Create MCI Dynamically from common spec and image.",
  "api_connection_id": "tumblebug_api",
  "swagger_yaml_endpoint": "/tumblebug/api/doc.yaml",
  "endpoint": "/ns/{nsId}/mciDynamic"
}

Examples of these JSONs can be found at: https://github.com/cloud-barista/cm-cicada/tree/main/lib/airflow/example/task_component

The Task Component automatically generated from the above JSON is as follows:

Expand

{
  "id": "796645fc-c594-4263-a9ff-243051d1f3a5",
  "name": "tumblebug_mci_dynamic",
  "description": "Create MCI Dynamically from common spec and image.",
  "data": {
    "options": {
      "api_connection_id": "tumblebug_api",
      "endpoint": "/tumblebug/ns/{nsId}/mciDynamic",
      "method": "POST",
      "request_body": "{\n    \"description\": \"Made in CB-TB\",\n    \"installMonAgent\": \"no\",\n    \"label\": {},\n    \"name\": \"mci01\",\n    \"systemLabel\": \"\",\n    \"vm\": [\n        {\n            \"commonImage\": \"ubuntu18.04\",\n            \"commonSpec\": \"aws+ap-northeast-2+t2.small\",\n            \"connectionName\": \"string\",\n            \"description\": \"Description\",\n            \"label\": {},\n            \"name\": \"g1-1\",\n            \"rootDiskSize\": \"default, 30, 42, ...\",\n            \"rootDiskType\": \"default, TYPE1, ...\",\n            \"subGroupSize\": \"3\",\n            \"vmUserPassword\": \"string\"\n        }\n    ]\n}"
    },
    "body_params": {
      "required": [
        "name",
        "vm"
      ],
      "properties": {
        "description": {
          "type": "string",
          "example": "Made in CB-TB"
        },
        "installMonAgent": {
          "type": "string",
          "description": "InstallMonAgent Option for CB-Dragonfly agent installation ([yes/no] default:no)",
          "default": "no",
          "enum": [
            "yes",
            "no"
          ],
          "example": "no"
        },
        "label": {
          "type": "object",
          "description": "Label is for describing the object by keywords"
        },
        "name": {
          "type": "string",
          "example": "mci01"
        },
        "systemLabel": {
          "type": "string",
          "description": "SystemLabel is for describing the mci in a keyword (any string can be used) for special System purpose",
          "example": ""
        },
        "vm": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "commonImage": {
                "type": "string",
                "description": "CommonImage is field for id of a image in common namespace",
                "example": "ubuntu18.04"
              },
              "commonSpec": {
                "type": "string",
                "description": "CommonSpec is field for id of a spec in common namespace",
                "example": "aws+ap-northeast-2+t2.small"
              },
              "connectionName": {
                "type": "string",
                "description": "if ConnectionName is given, the VM tries to use associtated credential.\nif not, it will use predefined ConnectionName in Spec objects"
              },
              "description": {
                "type": "string",
                "example": "Description"
              },
              "label": {
                "type": "object",
                "description": "Label is for describing the object by keywords"
              },
              "name": {
                "type": "string",
                "description": "VM name or subGroup name if is (not empty) && (> 0). If it is a group, actual VM name will be generated with -N postfix.",
                "example": "g1-1"
              },
              "rootDiskSize": {
                "type": "string",
                "description": "\"default\", Integer (GB): [\"50\", ..., \"1000\"]",
                "default": "default",
                "example": "default, 30, 42, ..."
              },
              "rootDiskType": {
                "type": "string",
                "description": "\"\", \"default\", \"TYPE1\", AWS: [\"standard\", \"gp2\", \"gp3\"], Azure: [\"PremiumSSD\", \"StandardSSD\", \"StandardHDD\"], GCP: [\"pd-standard\", \"pd-balanced\", \"pd-ssd\", \"pd-extreme\"], ALIBABA: [\"cloud_efficiency\", \"cloud\", \"cloud_essd\"], TENCENT: [\"CLOUD_PREMIUM\", \"CLOUD_SSD\"]",
                "default": "default",
                "example": "default, TYPE1, ..."
              },
              "subGroupSize": {
                "type": "string",
                "description": "if subGroupSize is (not empty) && (> 0), subGroup will be generated. VMs will be created accordingly.",
                "default": "1",
                "example": "3"
              },
              "vmUserPassword": {
                "type": "string"
              }
            }
          }
        }
      }
    },
    "path_params": {
      "required": [
        "nsId"
      ],
      "properties": {
        "nsId": {
          "type": "string",
          "description": "Namespace ID",
          "default": "default"
        }
      }
    },
    "query_params": {
      "properties": {
        "option": {
          "type": "string",
          "description": "Option for MCI creation",
          "enum": [
            "hold"
          ]
        }
      }
    }
  },
  "created_at": "2024-11-01T14:16:21.541043563+09:00",
  "updated_at": "2024-11-01T17:03:37.326385637+09:00",
  "is_example": true
}

SMTP

1. Add SMTP info

file path : /_airflow/docker-compose.yml

modify docker-compose.yml file and enter your smtp info.

gmail example : https://support.google.com/a/answer/176600?hl=en

...
    airflow-server:
        environment:
            AIRFLOW__SMTP__SMTP_HOST: 'smtp.gmail.com'
            AIRFLOW__SMTP__SMTP_USER: '[email protected]'
            AIRFLOW__SMTP__SMTP_PASSWORD: 'wtknvaprkkwyaurd'
            AIRFLOW__SMTP__SMTP_PORT: 587
            AIRFLOW__SMTP__SMTP_MAIL_FROM: '[email protected]'
...

2. Modify mail.py

file path : /_airflow/airflow-home/dags/mail.py

Modify the recipient's email address in the email_task.

...
    email_task = EmailOperator(
        task_id='send_email',
        to='Your [email protected]',
        subject='DAG 상태 보고서',
        ...
    )
...

3. Add taskComponent

Add trigger_email task component at the bottom of the workflow to receive email alarms.

...
         {
           "name": "trigger_email",
           "task_component": "trigger_email",
           "request_body": "",
           "path_params": {},
           "dependencies": [
             "{$Pre_taskName}"
           ]
         }

...

GET task log

1. GET workflow RunId

[GET] /workflow/{wfId}/runs

2. GET taskId and task_Try_Num

[GET] /workflow/{wfId}/workflowRun/{wfRunId}/taskInstances

3. GET execution task log

[GET] /workflow/{wfId}/workflowRun/{wfRunId}/task/{taskId}/taskTryNum/{taskTyNum}/logs

Health-check

Check if CM-Cicada is running

curl http://127.0.0.1:8083/cicada/readyz

# Output if it's running successfully
# {"message":"CM-Cicada API server is ready"}

Check out all APIs

Cicada APIs (Swagger Document)

Name		Name	Last commit message	Last commit date
Latest commit History 284 Commits
.github/workflows		.github/workflows
_airflow		_airflow
_data		_data
cmd/cm-cicada		cmd/cm-cicada
common		common
conf		conf
dao		dao
db		db
docs/api		docs/api
lib		lib
pkg/api/rest		pkg/api/rest
scripts		scripts
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.yaml		docker-compose.yaml
go.mod		go.mod
go.sum		go.sum

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Cloud Migration Workflow Management

Overview

Development environment

How to run

1. Build and run

2. Write configuration file

3. Check workflow template

4. Create the workflow

5. Run the workflow

About Task Component

SMTP

1. Add SMTP info

2. Modify mail.py

3. Add taskComponent

GET task log

1. GET workflow RunId

2. GET taskId and task_Try_Num

3. GET execution task log

Health-check

Check out all APIs

About

Releases 6

Packages

Contributors 6

Languages

License

cloud-barista/cm-cicada

Folders and files

Latest commit

History

Repository files navigation

Cloud Migration Workflow Management

Overview

Development environment

How to run

1. Build and run

2. Write configuration file

3. Check workflow template

4. Create the workflow

5. Run the workflow

About Task Component

SMTP

1. Add SMTP info

2. Modify mail.py

3. Add taskComponent

GET task log

1. GET workflow RunId

2. GET taskId and task_Try_Num

3. GET execution task log

Health-check

Check out all APIs

About

Resources

License

Stars

Watchers

Forks

Releases 6

Packages 0

Contributors 6

Languages

Packages