Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deployment Issues with Azure PowerShell version 13.0.0 #907

Open
1 task done
pejuborg opened this issue Nov 20, 2024 · 24 comments
Open
1 task done

Deployment Issues with Azure PowerShell version 13.0.0 #907

pejuborg opened this issue Nov 20, 2024 · 24 comments
Assignees
Labels
Area: Accelerator ⚡ Issues / PR's related to Accelerators Needs: Attention 👋 Needs attention from the maintainers Type: Bug 🪲 Something isn't working Type: Upstream Dependency ⬆️ something must happen before start something else

Comments

@pejuborg
Copy link

pejuborg commented Nov 20, 2024

How do I troubleshoot Azure PowerShell / Bicep deployment issues?

I'm struggling to get my Azure DevOps deployment of ALZ-Bicep to work, there are quite a few tweaks that must be done.

My latest hurdle is an error that I don't know how to find the root cause to.

When running the step "Built-in and Custom Policy Assignments Deployment" it fails with the error "Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject", and there is nothing that explains what causes this exception.

Code of Conduct

  • I agree to follow this project's Code of Conduct
@cspring86
Copy link

I've just upgraded to v0.20.0 and I'm now getting this error too.

I'm getting it on the following steps:

  • Custom Policy Definitions Deployment
  • Custom Management Group Diagnostic Settings
  • Built-in and Custom Policy Assignments Deployment

The first two steps fail, but eventually pass after a number of re-tries, however the last step never succeeds and hence fails the whole pipeline.

@cspring86
Copy link

To add, it looks like it's happening on the bicep build command.

If I run this locally with the same version of Bicep, it works perfectly.

@Chris031196
Copy link

Chris031196 commented Nov 20, 2024

I am encoutering this exact issue, too, but only since today.
Everything worked great yesterday, so something must have changed since yesterday.
Shouldn't be a v0.20.0 issue, since I have used that since around it was released...

@cspring86
Copy link

Maybe an Azure DevOps runner issue?

@jtracey93
Copy link
Collaborator

Is someone able to confirm the versions of bicep and the az powershell modules running on their hosted runners to help us troubleshoot further?

Might also be worth trying to pin the az powershell module version as per https://github.com/Azure/ALZ-Bicep/wiki/KnownIssues#issue-1-what-if-check-fails-within-azure-devops-pipelinegithub-actions-workflow-with-the-error-additional-content-found-in-json-reference-object-a-json-reference-object-should-only-have-a-ref-property-path-parresourcelockconfigdefaultvalue

@Chris031196 @cspring86 @pejuborg

Cc: @oZakari

@jtracey93 jtracey93 added Area: Accelerator ⚡ Issues / PR's related to Accelerators Needs: Author Feedback 👂 Needs the author to provide feedback Type: Bug 🪲 Something isn't working labels Nov 20, 2024
@cspring86
Copy link

My versions are:

  • AZ PowerShell Module: 13.0.0
  • Bicep: v0.31.92

@pejuborg
Copy link
Author

I've been able to test with earlier versions of Bicep with no luck. My suspicion is Az PowerShell 13.

@microsoft-github-policy-service microsoft-github-policy-service bot added Needs: Attention 👋 Needs attention from the maintainers and removed Needs: Author Feedback 👂 Needs the author to provide feedback labels Nov 20, 2024
@jtracey93
Copy link
Collaborator

V13.0.0 came out yesterday 🤷‍♂️ so I'd put my money there. https://www.powershellgallery.com/packages/Az/13.0.0

Can someone pin to an older version and retry. Say 12.4.0?

Following the guidance I shared in the previous comment

@jtracey93 jtracey93 added Needs: Author Feedback 👂 Needs the author to provide feedback and removed Needs: Attention 👋 Needs attention from the maintainers labels Nov 20, 2024
@pejuborg
Copy link
Author

Doing it as we "speak"!

The step "Check Az PowerShell Version" is not authored to work with other than the latest version, but the other PowerShell steps seem to.

@microsoft-github-policy-service microsoft-github-policy-service bot added Needs: Attention 👋 Needs attention from the maintainers and removed Needs: Author Feedback 👂 Needs the author to provide feedback labels Nov 20, 2024
@pejuborg
Copy link
Author

Pinning Az PS to 12.5.0 makes the step "Built-in and Custom Policy Assignments Deployment" succeed.

@cspring86
Copy link

Pinning the AZ pwsh module to 12.4.0 has worked for me.

However, with 13.0.0 on my local machine, it works as well; I can't re-create the issue locally. I've tried in bash and pwsh.

@oZakari oZakari self-assigned this Nov 21, 2024
@oZakari oZakari added the Type: Upstream Dependency ⬆️ something must happen before start something else label Nov 21, 2024
@oZakari oZakari pinned this issue Nov 21, 2024
@oZakari
Copy link
Contributor

oZakari commented Nov 21, 2024

Thank you, everyone, for reporting the issue and pitching in with troubleshooting—teamwork truly makes the dream work! For now, I’ve updated the Known Issues Wiki with all the information you’ve provided.

Starting tomorrow, I’ll dive deeper into troubleshooting this and will also open an issue with the relevant team(s) to get things moving.

As it stands, I'm wondering if it's associated to the ADO Azure PowerShell task considering that @cspring86 confirmed it worked fine locally with the latest version. That said, if anyone is using GitHub Actions for their pipelines and running into the same issue, please let me know!

@awebre
Copy link

awebre commented Nov 22, 2024

Hey y'all, we've just hit this same issue in GitHub Actions. Haven't tried to pin the Az-Powershell module yet, but I'm hoping that will fix it here too.

@jtracey93
Copy link
Collaborator

Hey @awebre, please let us know if it does 👍

@oZakari feeling like this may need to be a bug raised on the Az PowerShell module?

@awebre
Copy link

awebre commented Nov 22, 2024

@jtracey93 Pinning to 12.5.0 does seem to have resolved the issue.

@jtracey93
Copy link
Collaborator

@awebre Can you try 12.4.0 as that worked for another person in this thread; this'll help us narrow down the issue

@oZakari oZakari changed the title Azure DevOps deployment troubleshooting Deployment Issues with Azure PowerShell version 13.0.0 Nov 23, 2024
@oZakari
Copy link
Contributor

oZakari commented Nov 23, 2024

It seems this issue is specific to Azure PowerShell version 13.0.0, as I’ve only been able to replicate it with this version across all three deployment methods (Azure DevOps pipelines, GitHub Actions workflows, and local environments). Versions 12.5.0 and 12.4.0 do not appear to be affected.

An issue has been opened in the Azure PowerShell module repository: Azure PowerShell Issue #26752. Additionally, I’ve submitted an internal incident report to Microsoft for further investigation.

While I have only been able to replicate this with the ALZ Default Policy Assignments module, others have reported similar deployment failures with other modules. This may depend on specific customizations or parameter configurations. However, my suspicion is that the issue relates to JSON serialization in the ALZ Default Policy Modules, which we use to reference policy assignments.

I will provide updates in this thread as more information becomes available from the Azure PowerShell team.

@Chris031196
Copy link

Chris031196 commented Dec 3, 2024

Could anybody get this to work with self-hosted agents, too?
It worked great for me with AZ pwsh version 12.5.0 until I tried to use self-hosted agents today. I used the container instances which are created during the accelerator bootstrap phase.

With the not self-hosted agents, on the first deployment step it says in the console output:
Az version 12.5.0 not avaiable locally on the agent. Downloading dynamically.

With self-hosted agents, I just get this:

Exception: /azp/agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.248.3/Utility.ps1:94
Line |
  94 |  …             throw ("Could not find the module path with given version …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Could not find the module path with given version.

##[error]PowerShell exited with code '1'.

I even tried to add "Install-Module -Name Az -RequiredVersion 12.5.0" in the script for the "Check Az PowerShell Version" step, which did not make any difference...

@russelltomkins
Copy link

russelltomkins commented Dec 5, 2024

I am seeing the same "Could not find the module path with given version" error on the self-hosted Azure Container agents deployed by the bootstrap process, is there something else that must be updated in the container definitions as well?

@oZakari
Copy link
Contributor

oZakari commented Dec 6, 2024

@Chris031196 @russelltomkins - In .github\actions\bicep-installer\action.yaml, can you try adding the following Action and comment out the other tasks pertaining to Azure PowerShell:

- name: Install AZ PowerShell
  uses: azure/powershell@v2
  with:
    azPSVersion: 12.5.0
    inlineScript: |
      Write-Host "Installing AZ PowerShell version 12.5.0"

@Chris031196
Copy link

Chris031196 commented Dec 6, 2024

Alright, I will try it. I am using Azure DevOps, so my folder structure is a little different. I will add it to my .pipelines\helpers\bicep-installer.yaml.

EDIT: aah sorry, of course I cannot add this directly, as the Azure pipelines yaml structure is different.

So for me it should be smth like this maybe (?), which I already tried:

  - task: AzurePowerShell@5
    displayName: "Install AZ PowerShell"
    inputs:
      azureSubscription: ${{ parameters.serviceConnection }}
      pwsh: true
      azurePowerShellVersion: OtherVersion
      preferredAzurePowerShellVersion: 12.5.0
      ScriptType: "InlineScript"
      Inline: |
        Write-Host "Installing AZ PowerShell version 12.5.0"

@russelltomkins
Copy link

russelltomkins commented Dec 6, 2024

I added the following PowerShell task as the first task in ".\pipelines\helpers\bicep-installer.yaml" in the Templates repo and things are looking better now.

Warning: This task is very slow, my container instance took over 4 mins in whatif stage and 1.5 mins in deploy stage to complete.

  - task: PowerShell@2
    displayName: "Save Azure PowerShell 12.5.0 Module"
    inputs:
      targetType: "inline"
      script: |
        if(!(Test-Path -Path "/usr/share/az_12.5.0")) {
          mkdir "/usr/share/az_12.5.0"
          Save-Module -Name AZ -RequiredVersion 12.5.0 -path "/usr/share/az_12.5.0"
        }

@oZakari
Copy link
Contributor

oZakari commented Dec 6, 2024

Apologies, you both referenced self-hosted agents and I gave you a GitHub change. 🤦‍♂

Let me test that as well to see if there is another way apart from @russelltomkins's approach.

@oZakari
Copy link
Contributor

oZakari commented Dec 10, 2024

@russelltomkins is using the correct approach for the self-hosted agents. The Azure DevOps PowerShell task assumes the Az PowerShell module is saved in a specific non-standard directory, /usr/share/az_. By default, PowerShell installs modules to paths like $HOME/.local/share or /usr/local/share, which causes the task to fail if the module isn’t found. Thanks for sharing your solution @russelltomkins!

Also, looks like the Azure PowerShell team has a fix committed, we are just waiting on them to create a new release. Also, sounds like the bug has the potential to impact many different types of deployments and is non-deterministic which is why I wasn't able to replicate some of the deployment failures that others in this thread experienced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Accelerator ⚡ Issues / PR's related to Accelerators Needs: Attention 👋 Needs attention from the maintainers Type: Bug 🪲 Something isn't working Type: Upstream Dependency ⬆️ something must happen before start something else
Projects
None yet
Development

No branches or pull requests

7 participants