Install Windows HotFix using Terraform on AWS

Question

I have a very simple PowerShell script that uploads a generated test file to an AWS S3 bucket from a Windows 2008 R2 Datacenter server (clean AWS instance). If I run the script remotely on the server using Terraform (remote-exec provisioner), the script fails on the S3 upload with a StackOverflowException. When I run the script directly on the server, it runs fine and uploads the file.

I've experimented with different sizes for the file and 14.5MB seems to be about the maximum that works before the StackOverflowException occurs. Just about any size works fine when I RDP into the server and run the script directly. I've tested 200MB and it works fine.

Any idea why this is happening or what I can do to fix it? The actual file I need to upload is 50MB.

Here are the essential parts to recreate the problem. terraform.tf file:

resource "aws_instance" "windows" {
  count                       = "1"
  ami                         = "ami-e935fc94" #base win 2008 R2 datacenter
  instance_type               = "t2.micro"

  connection {
    type     = "winrm"
    user     = "<username>"
    password = "<password>"
    timeout  = "30m"
  }

  provisioner "file" {
    source      = "windows/upload.ps1"
    destination = "C:\\scripts\\upload.ps1"
  }

  provisioner "remote-exec" {
    inline = [
      "powershell.exe -File C:\\scripts\\upload.ps1"
    ]
  }
}

The PowerShell script is very simple. upload.ps1:

$f = new-object System.IO.FileStream C:\Temp\test.dat, Create, ReadWrite
$f.SetLength(40MB) # change this to 14.5MB and it works!
$f.Close()
Write-S3Object -BucketName "mybucket" -Folder "C:\Temp" -KeyPrefix "20180322" -SearchPattern "*.dat"

The error that I receive when launching the script from Terraform (remote-exec provisioner):

aws_instance.windows (remote-exec): Process is terminated due to StackOverflowException.

Running upload.ps1 from RDP on the server itself works fine, including larger files (tested up to 200MB).

Here is the version information:

Microsoft Windows Server 2008 R2 Datacenter
Powershell Version: 3.0
AWS Tools for Windows PowerShell, Version 3.3.245.0
Amazon Web Services SDK for .NET, Core Runtime Version 3.3.21.15

how do you manage to get a stackoverflow with powershell at all ? :) — 4c74356b41, Mar 22 '18 at 18:26
it actually makes it tricky googling the problem since "stackoverflow" isn't very specific because of this site — DarkerIvy, Mar 22 '18 at 18:57

DarkerIvy · Accepted Answer · 2018-03-31T03:13:31.910

This problem results from a Windows bug. This is all fine and good for a standard Windows server -- you can patch and move on. But, things are more tricky with AWS automation using Terraform.

The ideal solution would allow 1) use of the base AMI, 2) apply the hotfix to itself, and 3) then run the WinRM remote-exec, all from Terraform. Another solution would be to create an AMI with the hotfix installed and have Terraform generate instances using that AMI. However, then you're stuck maintaining AMIs.

Normally, I grab the Microsoft-provided base AMI using a filter:

data "aws_ami" "windows2008" {
  most_recent = true

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "name"
    values = ["Windows_Server-2008-R2_SP1-English-64Bit-Base*",]
  }

  owners = ["801119661308", "amazon"]
}

Then I use that AMI to create the AWS instance:

resource "aws_instance" "windows" {
  count                       = "1"
  ami                         = "${data.aws_ami.windows2008.id}"
  ...
}

But, the base AMI doesn't have the hotfix installed allowing you to avoid this WinRM/Windows bug. This is were it gets tricky.

You can use a userdata script to perform a multi-phase setup. In the first boot of the instance (Phase 1), we'll block the instance so that the remote-exec doesn't come in before we're ready. Then, we'll download and install the hotfix and we'll reboot (thanks to Niklas Akerlund, Micky Balladelli and Techibee). On the second boot (in method described here), we'll unblock the instance (enable WinRM) so that the remote-exec can connect.

Here's my userdata/PowerShell script:

$StateFile = "C:\Temp\userdata_state.txt"
If(-Not (Test-Path -Path $StateFile))
{
  # PHASE 1

  # Close the instance to WinRM connections until instance is ready (probably already closed, but just in case)
  Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth @{Basic=`"false`"}" -Wait

  # Set the admin password for WinRM connections
  $Admin = [adsi]("WinNT://./Administrator, user")
  $Admin.psbase.invoke("SetPassword", "${tfi_rm_pass}")

  # Create state file so after reboot it will know
  New-Item -Path $StateFile -ItemType "file" -Force

  # Make it so that userdata will run again after reboot
  $EC2SettingsFile="C:\Program Files\Amazon\Ec2ConfigService\Settings\Config.xml"
  $Xml = [xml](Get-Content $EC2SettingsFile)
  $XmlElement = $Xml.get_DocumentElement()
  $XmlElementToModify = $XmlElement.Plugins

  Foreach ($Element in $XmlElementToModify.Plugin)
  {
      If ($Element.name -eq "Ec2HandleUserData")
      {
          $Element.State="Enabled"
      }
  }
  $Xml.Save($EC2SettingsFile)

  # Download and install hotfix

  # Download self-extractor
  $DownloadUrl = "https://hotfixv4.trafficmanager.net/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix467402/7600/free/463984_intl_x64_zip.exe"
  $HotfixDir = "C:\hotfix"
  $HotfixFile = "$HotfixDir\KB2842230.exe"
  mkdir $HotfixDir
  (New-Object System.Net.WebClient).DownloadFile($DownloadUrl, $HotfixFile)

  # Extract self-extractor
  Add-Type -AssemblyName System.IO.Compression.FileSystem
  [System.IO.Compression.ZipFile]::ExtractToDirectory($HotfixFile, $HotfixDir)

  # Install - NOTE: wusa returns immediately, before install completes, so you must check process to see when it finishes
  Get-Item "$HotfixDir\*.msu" | Foreach { wusa ""$_.FullName /quiet /norestart"" ; While (@(Get-Process wusa -ErrorAction SilentlyContinue).Count -ne 0) { Start-Sleep 3 } }

  # Reboot
  Restart-Computer
}
Else 
{
  # PHASE 2

  # Open WinRM for remote-exec
  Start-Process -FilePath "winrm" -ArgumentList "quickconfig -q"
  Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service @{AllowUnencrypted=`"true`"}" -Wait
  Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth @{Basic=`"true`"}" -Wait
  Start-Process -FilePath "winrm" -ArgumentList "set winrm/config @{MaxTimeoutms=`"1900000`"}"
}

Install Windows HotFix using Terraform on AWS

1 Answers1