Write multi-line secret to windows runner in GitHub workflow

Question

Summary

What specific syntax must be changed in the code below in order for the multi-line contents of the $MY_SECRETS environment variable to be 1.) successfully written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file on a Windows runner in the GitHub workflow whose code is given below, and 2.) read by the simple Python 3 main.py program given below?

PROBLEM DEFINITION:

The echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.

We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. ... This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.

EXPECTED RESULT:

The multi-line secret should be printed into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file and read by main.py.

The resulting printout of the contents of the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file should look like:

***  
***  
***  
***

LOGS THAT DEMONSTRATE THE FAILURE:

The result of running main.py in the GitHub Actions log is:

ccc item is:  $MY_SECRETS

As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.

REPO FILE STRUCTURE:

Reproducing this error requires only 2 files in a repo file structure as follows:

.github/
    workflows/
        test.yml
main.py

WORKFLOW CODE:

The minimal code for the workflow to reproduce this problem is as follows:

name: write-secrets-to-file
on:
  push:
    branches:
      - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          pathlib.Path("C:\\Users\\runneradmin\\somedir\\").mkdir(parents=True, exist_ok=True)
          print('About to: echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
          output = subprocess.getoutput('echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
          print(output)
          os.chdir('D:\\a\\myRepoName\\')
          mycmd = "python myRepoName\\main.py"
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
            # returns None while subprocess is running
            retcode = p.poll() 
            line = p.stdout.readline()
            print(line)
            if retcode is not None:
              break

MINIMAL APP CODE:

Then the minimal main.py program that demonstrates what was actually written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file is:

with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
  for item in file:
    print('ccc item is: ', str(item))
    if "var1" in item:
      print("Found var1")

STRUCTURE OF MULTI-LINE SECRET:

The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:

var1:value1
var2:value2
var3:value3
var4:value4

These 4 lines should be what gets printed out when main.py is run by the workflow, though the print for each line should look like *** because each line is a secret.

I'm investigating why this `output = subprocess.getoutput('echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')` isn't working but I think doing `MY_SECRETS=os.getenv("MY_SECRETS")` and then doing `output =subprocess.getoutput('echo "{}" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml').format(MYSECRETS)` will work — tur1ng, Sep 08 '22 at 15:00
You should narrow down the problem and write a minimal test code. This is not useful(You didn't even specify the python version). — relent95, Sep 13 '22 at 11:33
@relent95 Two other users were able to meet the requirements specified by this OP. Also, this OP has had a python3 tag from the beginning. — CodeMed, Sep 14 '22 at 00:18

Frank · Answer 1 · 2022-09-13T07:53:22.197

The problem is - as it is so often - the quirks of Python with byte arrays and strings and en- and de-coding them in the right places...

Here is what I used:

test.yml:

name: write-secrets-to-file
on:
  push:
    branches:
    - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          import os
          # using os.path.expanduser() instead of hard-coding the user's home directory
          pathlib.Path(os.path.expanduser("~/somedir")).mkdir(parents=True, exist_ok=True)
          secrets = os.getenv("MY_SECRETS")
          with open(os.path.expanduser("~/somedir/mykeys.yaml"),"w",encoding="UTF-8") as file:
              file.write(secrets)
          mycmd = ["python","./main.py"]
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
              # returns None while subprocess is running
              retcode = p.poll()
              line = p.stdout.readline()
              # If len(line)==0 we are at EOF and do not need to print this line.
              # An empty line from main.py would be '\n' with len('\n')==1!
              if len(line)>0:
                # We decode the byte array to a string and strip the
                # new-line characters \r and \n from the end of the line,
                # which were read from stdout of main.py
                print(line.decode('UTF-8').rstrip('\r\n'))
              if retcode is not None:
                  break

main.py:

import os
# using os.path.expanduser instead of hard-coding user home directory
with open(os.path.expanduser('~/somedir/mykeys.yaml'),encoding='UTF-8') as file:
    for item in file:
        # strip the new-line characters \r and \n from the end of the line
        item=item.rstrip('\r\n')
        print('ccc item is: ', str(item))
        if "var1" in item:
            print("Found var1")

secrets.LIST_OF_SECRETS:

var1: secret1
var2: secret2
var3: secret3
var4: secret4

And my output in the log was

ccc item is:  ***
Found var1
ccc item is:  ***
ccc item is:  ***
ccc item is:  ***

tur1ng · Answer 2 · 2022-09-10T12:06:18.867

I tried the following code and it worked fine :

LIST_OF_SECRETS

key1:val1
key2:val2

Github action (test.yml)

name: write-secrets-to-file
on:
  push:
    branches:
      - main
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agentt
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import base64, subprocess, sys
          import os
          secrets = os.environ["MY_SECRETS"]
          
          def powershell(cmd, input=None):
              cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
              stdin = None if input is None else subprocess.PIPE
              process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
              if input is not None:
                  input = input.encode(sys.stdout.encoding)
              output, stderr = process.communicate(input)
              output = output.decode(sys.stdout.encoding).replace('\r\n', '\n')
              return output
          
          command = r"""$secrets = @'
          {}
          '@
          $secrets | Out-File -FilePath .\mykeys.yaml""".format(secrets)
          
          command1 = r"""Get-Content -Path .\mykeys.yaml"""
          
          powershell(command)
          print(powershell(command1))

Output

***
***

As you also mention in the question, Github will obfuscate any printed value containing the secrets with ***

EDIT : Updated the code to work with multiple line secrets. This answer was highly influenced by this one

Okay, this doesn't work with multiple line secrets. Seems like a powershell issue fixing it now. — tur1ng, Sep 09 '22 at 20:28
@CodeMed I updated the answer to work for multiple lines. Yes it may sound a bit weird having to encode the powershell commands into base64 before executing them, but this solved all the formatting/encoding issues between Python and Powershell that I encondered trying to use powershell's here string. As to why we are stating powershell in the command, we have to because we are overriding Action's default shell with Python and then the shell it opens is cmd. — tur1ng, Sep 09 '22 at 22:30
Your code does not call `main.py` as specified in the OP. Your code breaks when we use it to call `main.py` — CodeMed, Sep 11 '22 at 00:06

duthils · Accepted Answer · 2022-09-13T01:56:06.567

Edit: updated with fixed main.py and how to run it.

You can write the key file directly with Python:

      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
          with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

To avoid newline characters in your output, you need a main.py that removes the newlines (here with .strip().splitlines()):

main.py

with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
    for item in file.read().strip().splitlines():
        print('ccc item is: ', str(item))
        if "var1" in item:
            print("Found var1")

Here's the input:

LIST_OF_SECRETS = '
key:value
key2:value
key3:value
'

And the output:

ccc item is:  ***
Found var1
ccc item is:  ***
ccc item is:  ***
ccc item is:  ***

Here is my complete workflow:

name: write-secrets-to-file
on:
  push:
    branches:
      - master
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import os
          import pathlib
          pathlib.Path('C:\\Users\\runneradmin\\somedir\\').mkdir(parents=True, exist_ok=True)
          with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml', 'w') as key_file:
            key_file.write(os.environ['MY_SECRETS'])
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

Also, a simpler version using only Windows shell (Powershell):

      - name: Create key file
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          mkdir C:\\Users\\runneradmin\\somedir
          echo "$env:MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml
      - uses: actions/checkout@v3
      - name: Run main
        run: python main.py

George · Answer 4 · 2022-09-11T07:07:14.287

0

You need to use yaml library:

import yaml

data = {'MY_SECRETS':'''
var1:value1
var2:value2
var3:value3
var4:value4
'''}#add your secret 

with open('file.yaml', 'w') as outfile: # Your file
    yaml.dump(data, outfile, default_flow_style=False)

This is result: I used this.

edited Sep 11 '22 at 07:07

answered Sep 08 '22 at 15:07

George

40
12

This works only if you have access to `C:\\Users\\runneradmin\\somedir\\mykeys.yaml`. – George Sep 08 '22 at 15:10
Can you please test your code on an actual GitHub workflow calling a runner with `windows-latest` as the OP specifies? Your code does not work in the environment specified in the OP. It only takes a few minutes to set up the minimal example from the OP in a new GitHub repository if you have a free GitHub account. – CodeMed Sep 09 '22 at 19:01

Write multi-line secret to windows runner in GitHub workflow

Summary

4 Answers4

test.yml:

main.py:

secrets.LIST_OF_SECRETS: