Placement and unit changes causes removal

[hloeung]

Description

Updating units and placement causes application to be removed and re-created.

My Terraform plan is as follows:

resource "juju_application" "ntp" {
  name  = "ntp"
  model = local.juju_model_name

  constraints = "arch=amd64 cores=2 mem=4096M root-disk=20480M"

  charm {
    name     = "chrony"
    channel  = "latest/edge"
    revision = 34
    base     = "[email protected]"
  }

  config = {
    sources          = <<-EOT
      ntp://clock.isc.org?iburst=true
    EOT
  }

  units = 1
  placement = "0"

  expose {}
}

I changed units to 2 and placement to "0,1". Unfortunately, that caused the ntp application to be destroyed and a new one recreated. This removed by original unit and manually provisioned machine (because Juju doesn't support provisioning of OpenStack SR-IOV / OVN VMs).

prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ terraform apply
data.vault_generic_secret.juju_controller_certificate: Reading...
data.vault_generic_secret.juju_credentials: Reading...
data.vault_generic_secret.juju_credentials: Read complete after 0s [id=secret/prodstack5/roles/prod-ntp-nts-ps5/juju]
data.vault_generic_secret.juju_controller_certificate: Read complete after 0s [id=secret/prodstack5/juju/common/ca_certs/juju-controller-35-production-ps5]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # juju_application.grafana_agent will be created
  + resource "juju_application" "grafana_agent" {
      + constraints = (known after apply)
      + id          = (known after apply)
      + model       = "prod-ntp-nts-ps5"
      + name        = "grafana-agent"
      + placement   = (known after apply)
      + principal   = (known after apply)
      + storage     = (known after apply)
      + trust       = false
      + units       = 0

      + charm {
          + base     = "[email protected]"
          + channel  = "latest/edge"
          + name     = "grafana-agent"
          + revision = 260
          + series   = (known after apply)
        }
    }

  # juju_application.ntp will be created
  + resource "juju_application" "ntp" {
      + config      = {
          + "sources" = <<-EOT
                ntp://ntp.inria.fr?iburst=true,
                ntp://ntp-p1.obspm.fr?iburst=true,
                ntp://ntp1.time.nl?iburst=true,
                ntp://ntp0.fau.de?iburst=true,
                ntp://ntp-galway.hea.net?iburst=true,
                ntp://ntp1.inrim.it?iburst=true,
                ntp://ntp.se?iburst=true,
                ntp://ntp.atomki.mta.hu?iburst=true,
                ntp://time.apple.com?iburst=true,
                ntp://time-ios.apple.com?iburst=true,
                ntp://clock.uregina.ca?iburst=true,
                ntp://time-a.timefreq.bldrdoc.gov?iburst=true,
                ntp://clock.isc.org?iburst=true
            EOT
        }
      + constraints = "arch=amd64 cores=2 mem=4096M root-disk=20480M"
      + id          = (known after apply)
      + model       = "prod-ntp-nts-ps5"
      + name        = "ntp"
      + placement   = "0"
      + principal   = (known after apply)
      + storage     = (known after apply)
      + trust       = false
      + units       = 1

      + charm {
          + base     = "[email protected]"
          + channel  = "latest/edge"
          + name     = "chrony"
          + revision = 34
          + series   = (known after apply)
        }

      + expose {}
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

juju_application.grafana_agent: Creating...
juju_application.ntp: Creating...
juju_application.grafana_agent: Creation complete after 1s [id=prod-ntp-nts-ps5:grafana-agent]
juju_application.ntp: Creation complete after 1s [id=prod-ntp-nts-ps5:ntp]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ jsft
Model             Controller                         Cloud/Region           Version  SLA          Timestamp
prod-ntp-nts-ps5  juju-controller-35-production-ps5  prodstack5/prodstack5  3.5.3    unsupported  20:31:28Z

App            Version  Status   Scale  Charm          Channel      Rev  Exposed  Message
grafana-agent           unknown      0  grafana-agent  latest/edge  260  no
ntp                     waiting    0/1  chrony         latest/edge   34  yes      agent initialising

Unit    Workload  Agent       Machine  Public address          Ports  Message
ntp/0*  waiting   allocating  0        ntp-nts-ps5-1.internal         agent initialising

Machine  State    Address                 Inst id                        Base          AZ  Message
0        started  ntp-nts-ps5-1.internal  manual:ntp-nts-ps5-1.internal  [email protected]      Manually provisioned machine
1        started  ntp-nts-ps5-2.internal  manual:ntp-nts-ps5-2.internal  [email protected]      Manually provisioned machine
2        started  ntp-nts-ps5-3.internal  manual:ntp-nts-ps5-3.internal  [email protected]      Manually provisioned machine
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ 

Now plan updated with the only change to unit and placement:

prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ vi main.tf
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ terraform plan
data.vault_generic_secret.juju_credentials: Reading...
data.vault_generic_secret.juju_controller_certificate: Reading...
data.vault_generic_secret.juju_controller_certificate: Read complete after 0s [id=secret/prodstack5/juju/common/ca_certs/juju-controller-35-production-ps5]
data.vault_generic_secret.juju_credentials: Read complete after 0s [id=secret/prodstack5/roles/prod-ntp-nts-ps5/juju]
juju_application.grafana_agent: Refreshing state... [id=prod-ntp-nts-ps5:grafana-agent]
juju_application.ntp: Refreshing state... [id=prod-ntp-nts-ps5:ntp]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # juju_application.ntp must be replaced
-/+ resource "juju_application" "ntp" {
      ~ id          = "prod-ntp-nts-ps5:ntp" -> (known after apply)
        name        = "ntp"
      ~ placement   = "0" -> "0,1" # forces replacement
      + principal   = (known after apply)
      + storage     = (known after apply)
      ~ units       = 1 -> 2
        # (4 unchanged attributes hidden)

      ~ charm {
            name     = "chrony"
          ~ series   = "noble" -> (known after apply)
            # (3 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ terraform apply
data.vault_generic_secret.juju_credentials: Reading...
data.vault_generic_secret.juju_controller_certificate: Reading...
data.vault_generic_secret.juju_credentials: Read complete after 0s [id=secret/prodstack5/roles/prod-ntp-nts-ps5/juju]
data.vault_generic_secret.juju_controller_certificate: Read complete after 0s [id=secret/prodstack5/juju/common/ca_certs/juju-controller-35-production-ps5]
juju_application.ntp: Refreshing state... [id=prod-ntp-nts-ps5:ntp]
juju_application.grafana_agent: Refreshing state... [id=prod-ntp-nts-ps5:grafana-agent]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # juju_application.ntp must be replaced
-/+ resource "juju_application" "ntp" {
      ~ id          = "prod-ntp-nts-ps5:ntp" -> (known after apply)
        name        = "ntp"
      ~ placement   = "0" -> "0,1" # forces replacement
      + principal   = (known after apply)
      + storage     = (known after apply)
      ~ units       = 1 -> 2
        # (4 unchanged attributes hidden)

      ~ charm {
            name     = "chrony"
          ~ series   = "noble" -> (known after apply)
            # (3 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

juju_application.ntp: Destroying... [id=prod-ntp-nts-ps5:ntp]
juju_application.ntp: Destruction complete after 1s
juju_application.ntp: Creating...
╷
│ Error: Client Error
│
│   with juju_application.ntp,
│   on main.tf line 33, in resource "juju_application" "ntp":
│   33: resource "juju_application" "ntp" {
│
│ Unable to create application, got error: cannot add application "ntp": application already exists
╵
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$

Machine 0 is now removed from the model:

prod-ntp-nts-ps5@is-bastion-ps5:~/plan$ jsft
Model             Controller                         Cloud/Region           Version  SLA          Timestamp
prod-ntp-nts-ps5  juju-controller-35-production-ps5  prodstack5/prodstack5  3.5.3    unsupported  20:32:41Z

App            Version  Status   Scale  Charm          Channel      Rev  Exposed  Message
grafana-agent           unknown      0  grafana-agent  latest/edge  260  no

Machine  State    Address                 Inst id                        Base          AZ  Message
1        started  ntp-nts-ps5-2.internal  manual:ntp-nts-ps5-2.internal  [email protected]      Manually provisioned machine
2        started  ntp-nts-ps5-3.internal  manual:ntp-nts-ps5-3.internal  [email protected]      Manually provisioned machine
prod-ntp-nts-ps5@is-bastion-ps5:~/plan$

Good thing is that this is a new service but would cause issues when we want to scale out adding more units.

Urgency

Casually reporting

Terraform Juju Provider version

0.13.0

Terraform version

v1.7.2-dev

Juju version

3.5.3

Terraform Configuration(s)

No response

Reproduce / Test

terraform apply
(edit main.tf bumping units and updating placement)
terraform apply

Debug/Panic Output

No response

Notes & References

No response

[hloeung]

units = 1
placement = "0"

Changed to:

units = 2
placement = "0,1"

Expected behavior is for a new ntp unit to be provisioned to machine 1. The original unit on machine 0 remains.

[Aflynn50]

This is because the RequiresReplaceIfConfigured flag is on the placement directive. The placement directive can be complex and there are edge cases that will require replacement.

Would be good to get @hmlanigan's opinion on this.

[hmlanigan]

Yes, this is expected behavior due to how placement is defined in the schema.

[hmlanigan]

Duplicate of #512