verify_shared_folders fails even when powershell returns exitcode=>0

[deluxebrain]

vagrant up is failing for me with the following error:

Shared folders not properly configured. This is generally resolved by a 'vagrant halt && vagrant up'

Running with --debug Im getting the following log entries:

INFO dsc: Checking for shared folder: /tmp/vagrant-dsc-6/dsc/manifests/

This executes the following script:

DEBUG winrmshell: powershell executing:
$ProgressPreference='SilentlyContinue';
            $p = "/tmp/vagrant-dsc-6/dsc/manifests/"
            if ((Test-Path $p) -and (get-item $p).PSIsContainer) {
              exit 0
            }
            exit 1

if ($?) { exit 0 } else { if($LASTEXITCODE) { exit $LASTEXITCODE } else { exit 1 } }

Which in turns returns with a zero exit code:

DEBUG winrmshell: Output: {:data=>[{:stderr=>"#< CLIXML\r\n"}, {:stderr=>"<Objs Version=\"1.1.0.1\" xmlns=\"http://schemas.microsoft.com/powershell/2004/04\"><Obj S=\"progress\" RefId=\"0\"><TN RefId=\"0\"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N=\"SourceId\">1</I64><PR N=\"Record\"><AV>Preparing m"}, {:stderr=>"odules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S=\"progress\" RefId=\"1\"><TNRef RefId=\"0\" /><MS><I64 N=\"SourceId\">2</I64><PR N=\"Record\"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>"}], :exitcode=>0}

However - this is then immediately followed by an error that states that the shared folders cannot be found:

ERROR warden: Error occurred: Shared folders not properly configured. This is generally resolved by a 'vagrant halt && vagrant up'

I have manually verified that the shared folder as specified in the log file exists on the guest. I've also manually run the powershell script on the host and its returning true.

Working through your (most excellent!) three-part blog series about DSC provisioning, the only differences I can see between my packer builds and your examples are the following:

1. I've bumped the PowerShell version to 5 (by not specifying a version number in the choco install call).
2. I'm not brining in the SEEK PowerShell modules as these appear to have all been mainlined into the official PowerShell repository at this stage.

Any ideas?

Many thanks.

[mefellows]

Looks like a bug, thanks for raising.

[deluxebrain]

Thanks for the reply. I've spent some time digging into the issue and I think the problem actually lies with the vagrant winrm plugin and/or the exact combinations of powershell tooling related installs I've made on the packer image. This appears to lead to the situation where powershell calls over winrm returns the exit code in a garbled mess of xml.

I'll try to get some time to dig further - but at this stage I really don't actually think this directly an issue with your vagrant-dsc plugin.

Cheers

[dragon788]

@deluxebrain I'm curious what the WinRM settings of the box you are using are, memory limits, process limits, authentication=simple etc or are you using the default mfellows/win2012r2?

[deluxebrain]

@dragon788 Im using an AWS Windows Server 2012r2 AMI, setting up winrm in the aws user-data file as follows:
MaxMemoryPerShellMB=1024
Authentication=Basic
The rest Im assuming just defaults to "sensible values".
Cheers

[mefellows]

Interesting - are you using Vagrant DSC with the AWS plugin?

[deluxebrain]

Apologies - I've moved onto performing the DSC provisioning within Packer (using your pcaker-dsc plugin) and at the same time moved to AWS as the virtualisation layer. This is all working perfectly. My reply above re AWS was in error.

My issue with vagrant-dsc plugin is with vagrant images build out using packer and the vbox builder (as per blog 1 in your series) and using Windows Server 2012 R2 images downloaded from Microsoft Tech Bench.

I'll try to get some time over the weekend to strip back the packer builds to the bare minimum to confirm if the issue is with the various additional installs Im performing.

Interestingly, I've stumbled across this discussion that appears to be a variant of the issue I'm facing. The route cause in that case appears to be an interaction with an installed SSH-agent and the returned XML from winrm calls. Looking through my packer build script I've left in the installation of openssh - so I'll strip that out and retry.

Cheers

[mefellows]

I recall there being some weirdness when implementing this. It's possible there was a change in Vagrant or an upstream project that has meant this implementation has been impacted, but i've seen this before. Let me dig a bit further and come back to you.

P.S. Also, thanks for the kind words above! I was responding to a few issues on my mobile and was a bit hasty!

[mefellows]

I'm hoping the installed SSH isn't conflicting and am interested to hear back.

Looking at the output from above (the XML-ie stuff), that is all standard winrm XML.

I think the culprit is https://github.com/mitchellh/vagrant/blob/master/plugins/communicators/winrm/communicator.rb#L168 coupled with something in the operating environment (new PoSH setup, for instance). If you notice, even if exit code is 0, it checks for the presence (or lack-thereof) of any output to stderr. As you can see above, we get (expected) output to stderr and this is likely the cause of failure.

Will continue to investigate.

[mefellows]

OK, this was only committed 3 days after the initial DSC plugin release! I'd say this is likely part of the problem.

[mefellows]

@deluxebrain can you please confirm if the issue happens after the first time, if you re-provision the box (with vagrant provision --provision-with dsc) after a reboot (e.g. vagrant reload)?

[deluxebrain]

@mefellows Thanks for taking a look at this today and posting your thoughts - very interesting to follow along.
Just gone through the vagrant up; vagrant reload; vagrant provision --debug --provision-with dsc steps you list above and its giving the same error (i.e. the exit code of 0 combined with the presence of a stderr as you noted above).

[mefellows]

No worries. I've pushed a new version (v1.0.8). Do you mind upgrading and testing to see if this has fixed the issue? (Based on your comment just now it probably won't). I'll take a look again a bit later.

[deluxebrain]

Thanks. As you expected - just re-ran the above steps using v1.0.8 and same error.

[mefellows]

Thanks Matt, back to the drawing board! Can you please confirm the guest os
that you're running and any provisioning performed on it before vagrant
gets a hold of it? Also any Vagrantfiles or Packer scripts if you can share?
On 28 Jun 2016 7:29 am, "Matt Caton" [email protected] wrote:

Thanks. As you expected - just re-ran the above steps using v1.0.8 and
same error.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AADSjIteGy85ys1J1uIivrj_AhyzkXITks5qQECygaJpZM4I6qT0
.

[deluxebrain]

Absolutely.
Its all currently private repos stuff. I will factor our the core stuff into some gists and share it. Gimme a day or two.

[mefellows]

Thanks Matt, no worries.

On Wed, Jun 29, 2016 at 7:53 PM, Matt Caton [email protected]
wrote:

Absolutely.
Its all currently private repos stuff. I will factor our the core stuff
into some gists and share it. Gimme a day or two.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AADSjAKXV0iuItLKi4OOo7HgwYsHWsGMks5qQkCUgaJpZM4I6qT0
.

Matt Fellows

• http://www.onegeek.com.au/

[deluxebrain]

@mefellows Hey Matt. Took a little while but finally got around to pulling out the packer and vagrant configuration into a separate repos.

This contains:

1. The packer builder raw that builds out a vagrant image ready for Vagrant to provision via your vagrant-dsc plugin
2. An additional builder called baked that does the same step as Vagrant but using your packer-dsc plugin.

As before - the Vagrant provisioning is failing, but the packer-dsc provisioning is working beautifully (and is now being used in production - many thanks for the wonderful work).

This is very much cut down - but the issue still remains. However - stripping this all down has highlighted an issue with my packer builds in that the Window updates aren't completing. I had not noticed this before as the bulk of the work I'm doing is against AWS where obviously this is not a concern. So - I'm wondering if the issue is that the packer vagrant image is missing some critical Windows update.

I'll work on getting the Windows updates to go through, rerun the test, and get back with any updates as and when I have them.

Cheers!

[mefellows]

Awesome work Matt, I should be able to look at this tomorrow night and hopefully can get to the bottom of it. Good point around updates. My head has been out of this game for a little bit so hopefully with this it will trick me back into taming Windows :)

Thanks for the kinds words!

[deluxebrain]

Windows updates look to all be running in - now that I've moved them to Boxstarter and .... set the winrm timeout to 24 hours!

Unfortunately can't find a publicly url addressable Windows iso that is anywhere near fully patched...

[dragon788]

I'd suggest taking a look at MWrock's Packer templates if you want to use
boxstarter for really easy application of Windows updates and it handles
the reboots as well. The main difference is he applies them via
unattend.xml so that the reboots don't cause packer to move onto the next
provisioner.
On Jul 4, 2016 3:53 AM, "Matt Fellows" [email protected] wrote:

Awesome work Matt, I should be able to look at this tomorrow night and
hopefully can get to the bottom of it. Good point around updates. My head
has been out of this game for a little bit so hopefully with this it will
trick me back into taming Windows :)

Thanks for the kinds words!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAdxXsHJRSRoQTZxZ6Cbfi8tgUVh5t-4ks5qSMoDgaJpZM4I6qT0
.

[deluxebrain]

@dragon788 Yep - spent a good proportion of my weekend trying to get it to work as part of a packer provisioning step! You mentioned a few comments up about Matt Wrock using Boxstarter as part of the autounattend (which I had completely missed) and re-reading that article everything snapped into place. Cheers for help - appreciated.

[deluxebrain]

@mefellows nearly a day later and the Windows updates have all run in successfully.

However - looks like I've mucked up the Vagrant shared folder setup as part of removing rsync ( as part of simplifying the build ). I'll get that sorted out - and until I have its probably not worth you looking at the issue (don't want to waste any of your time)

Cheers

[mefellows]

No worries Matt - yeah those updates can be sloooow. Keep us in the loop, I'm keen to see where we get to with this.

[dragon788]

@mefellows I definitely think this is a combination of PowerShell 5 and the WinRM(fs) gems. I was getting a straight up failure of any shell provisioning step after I updated to PowerShell 5 because it spits out "Preparing modules for first use" which must fall to stderr since it probably isn't written to the stdout stream but rather to the debug stream with Write-Debug, but WinRM doesn't support that stream natively (see @mwrock's excellent article about a new WinRb feature in development, http://www.hurryupandwait.io/blog/a-look-under-the-hood-at-powershell-remoting-through-a-ruby-cross-plaform-lens) and this is causing the failures. I "cheated" to get past the initial error with shell scripts by bumping the version of winrm-fs to 0.4.3 (which seems to have a fix to partially ignore this message), but it still fails due to the issues listed above. It seems for the moment PowerShell 5 is incompatible with the DSC plugin without a bit of extra exception handling or parsing out the above message mentioned by @deluxebrain .

[mefellows]

Great debugging work there @dragon788! Hopefully this is something we can handle gracefully within the plugin otherwise a patch to WinRM ruby might be the sln.

[dragon788]

And the plot thickens. It appears that removing the Chocolatey generated PowerShell profile actually works around this issue. We'll see if it blows up when there is another progress bar generated, but at the very least this works around the "Preparing modules for first use" that was popping up and breaking things. Chocolatey also has a note about fixing this by not adding a default profile if it isn't there. chocolatey/choco#833

I should also note I've done the winrb "hack" recommended by Stefan to bump the version from 0.3.x to 0.4.x here, hashicorp/vagrant#7476 (comment) . This has gotten me at least somewhat working.

[mwrock]

Just stumpled on this thread...I have definitely suffered much pain from the "Preparing modules for first use" issue. The fix is usually to put $ProgressPreference = 'SilentlyContinue' first thing in the script. The winrm gem injects this for you which solves most cases. The winrm-v2 gem will be able to parse the separate powershell streams more cleanly and will never leak the progress stream into stderr (which is what is happening here).

[dragon788]

@mwrock Can't wait for v2, hopefully Hashicorp gets it pulled into Vagrant sooner than later. It appears from #7476 I referenced above that it has taken them a while to update previously.

[mwrock]

Yes I will pull it in shortly after its released.

[mefellows]

Thanks guys, so @dragon788 have you attempted the fix @mwrock suggested?

If three Matt's and a dragon can't solve this then there is no hope for humanity!

[dragon788]

I haven't tried with the silently proceed yet, my OCD made me fix a spot where a couple lines were blurring together and then I headed out of the office for a week of vacation. I may take a look while I'm on the road to see if it fixes the issue with the Chocolatey profile in place, but so far without the profile it has been doing all the DSC things pretty well (despite me doing weird stuff like partial configs that isn't supported by the plugin that I can tell yet).

[mefellows]

No worries @dragon788, I'll take a look at the PR shortly. Enjoy the break :-)

[ferventcoder]

Choco 0.10.0 fixed the issue of creating a profile.

[mefellows]

Will give the profile a shot - thanks all!