Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix launch time at small instances #84

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

valVk
Copy link
Contributor

@valVk valVk commented Jan 2, 2023

If select a smaller instance than t3.xlarge than jenkins does not have enough time to startup before systemctl will try to restart jenkins.service

This pull request add additional variable to have control under the timeout before systemctl will restart jenkins with resolution
systemd: jenkins.service start operation timed out. Terminating.

How to reproduce issue:

  • Set instance type for master as t3.small

How to test:

  • Checkout to this PR
  • Set variable time_to_start = 900 #15 mins

Example

provider "aws" {
  region  = var.region
  profile = var.profile
  default_tags {
    tags = {
      Environment = var.environment
      Owner       = "terraform"
      Project     = var.profile
      terraform   = "true"
    }
  }
}

data "terraform_remote_state" "remote" {
  backend = "s3"

  config = {
    profile = var.profile
    region  = var.region
    bucket  = "terraform-state"
    key     = "developemnt/terraform.tfstate"
  }
}

data "terraform_remote_state" "route53" {
  backend = "s3"

  config = {
    profile = var.profile
    region  = var.region
    bucket  = "terraform-state"
    key     = "global/route53/terraform.tfstate"
  }
}

locals {
  tags = {
    # contact     = var.contact
    # environment = var.environment
  }
}

module "jenkins_ha_agents" {
  # source          = "neiman-marcus/jenkins-ha-agents/aws"
  source          = "github.com/ValeriyDP/terraform-aws-jenkins-ha-agents"

  admin_password           = var.admin_password == "" ? random_password.admin_password[0].result : var.admin_password
  agent_max                = var.agent_max
  agent_min                = var.agent_min
  agent_volume_size        = var.agent_volume_size
  scale_down_number        = -1
  scale_up_number          = 1
  key_name                 = data.terraform_remote_state.remote.outputs.key_pair_name
  instance_type_controller = ["t3a.small"]
  instance_type_agents     = ["t3.xlarge", "t3a.xlarge"]

  time_to_start = 60 * 60

  custom_plugins = templatefile("init/custom_plugins.cfg", {})

  bastion_sg_name = data.terraform_remote_state.remote.outputs.bastion_sg[0].name
  domain_name     = "${data.terraform_remote_state.route53.outputs.dns_zone_prefix}."

  private_subnet_name = "${var.environment}-${var.project_name}-private-*"
  public_subnet_name  = "${var.environment}-${var.project_name}-public-*"

  r53_record = "ci.${data.terraform_remote_state.route53.outputs.dns_zone_prefix}"
  region     = var.region

  ssl_certificate = data.terraform_remote_state.route53.outputs.dns_zone_prefix
  ssm_parameter   = "/dev/ci/jenkins"

  tags     = local.tags
  vpc_name = "${var.environment}-${var.project_name}"
}

resource "random_password" "admin_password" {
  count       = var.admin_password == "" ? 1 : 0
  length      = 12
  special     = false
  lower       = true
  numeric     = true
  min_numeric = 4
  min_lower   = 4
  min_special = 0
  min_upper   = 4
}

Links for reference:

@BrettKorp
Copy link

BrettKorp commented Mar 19, 2023

For what it's worth I took the logic from this PR and implemented it outside of the module using the extra_master_userdata argument. It would be nice to get this implemented into the next module version though. +1 For this PR.

This works for me using a t3a.large for the primary. Thanks for this!

My Module Instantiation looks like this:

module "jenkins_cluster" {
  source  = "neiman-marcus/jenkins-ha-agents/aws"
  ...
  extra_master_userdata       = templatefile("extra-master-userdata.cfg", {time_to_start=300})
  extra_master_userdata_merge = "list(prepend)+dict(recurse_array)+str()"
}

And my extra-master-userdata.cfg template file looks like this:

---
runcmd:
    - sudo mkdir /etc/systemd/system/jenkins.service.d
    - echo -e "[Service]\nTimeoutStartSec=${time_to_start}" | sudo tee /etc/systemd/system/jenkins.service.d/startup-timeout.conf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants