0

I have a Python script named app.py that has the value of the project ID, 

project_id = "p007-999"

I hard code it inside the .gitlab-ci.yml file provided below, 

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - export WE_PROJECT_ID="p007-999"
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

I would like to automate this. I think the steps for that will be,

a. write the project_id value from the Python script to a shell script variables.sh.
b. In the before_script: of the YML file, execute the variables.sh and read the value from there.

How do I achieve it correctly?

Anthon
  • 69,918
  • 32
  • 186
  • 246
Arefe
  • 11,321
  • 18
  • 114
  • 168
  • @AliBenZarrouk its mention in the first line of the question. I have a python file `app.py` where the value is originally generated and acquired. – Arefe Apr 06 '19 at 08:08
  • Gitlab uses the extension `.yml` although the recommended extension has been `.yaml` since Sep 2006 (see the FAQ on yaml.org). The file format acronym always has been YAML and never YML – Anthon Apr 06 '19 at 08:27
  • @Arefe do you want the content of this `variables.sh` script shared between all the stages ? Or do you just want to `project_id` variable accessible to the `build` stage ? – Ali Ben Zarrouk Apr 06 '19 at 08:42

2 Answers2

2

You can do this with ruamel.yaml, which was specfically developed to do these kind of round-trip updates (disclaimer: I am the author of that package).

Assuming your input is:

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - PID_TO_REPLACE
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

And your code something like:

import sys
from pathlib import Path
import ruamel.yaml


def update_project_id(path, pid):
   yaml = ruamel.yaml.YAML()
   yaml.indent(sequence=4, offset=2) # non-standard indent of 4 for sequences
   yaml.preserve_quotes = True
   data = yaml.load(path)

   data['before_script'][0] = 'export WE_PROJECT_ID="' + pid + '"'
   yaml.dump(data, path)

file_name = Path('.gitlab-ci.yml')
project_id = "p007-999"

update_project_id(file_name, project_id)

which gives as output:

# list of enabled stages, the default should be built, test, publish
stages:
  - build
  - publish

before_script:
  - export WE_PROJECT_ID="p007-999"
  - docker login -u "$WELANCE_REGISTRY_USER" -p "$WELANCE_REGISTRY_TOKEN" registry.welance.com

build:
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_HOST: docker:2375
  script:
    - echo $WE_PROJECT_ID
    - cd templates && pwd && yarn install && yarn prod && cd ..
    - docker build -t registry.welance.com/$WE_PROJECT_ID:$CI_COMMIT_REF_SLUG.$CI_COMMIT_SHA -f ./build/ci/Dockerfile .

(including the comment, which you lose when using by most other YAML loader/dumpers)

Anthon
  • 69,918
  • 32
  • 186
  • 246
  • My yaml is generated from the `jinja2` template and I get the error like `ruamel.yaml.scanner.ScannerError: while scanning for the next token found character '\t' that cannot start any token in ".gitlab-ci.yml", line 55, column 1`. However, I tested with a valid `yaml` file and seems your code is working. – Arefe Apr 06 '19 at 11:46
  • You have a TAB character in your jinja2 template, make sure you expand those to the appropriate amount of spaces – Anthon Apr 06 '19 at 12:22
  • Okay how do I properly format the Jinja2 template? Is there any online formattor available? – Arefe Apr 06 '19 at 12:29
  • There is no jinja2 format, it is a templating language that can generate any (text) file you want. Just take an editor where you can see the tabs and delete them then insert spaces. You can do **a line at a time** and run the template expansion + above program to make sure you did not introduce YAML errors, until you are confident about your changes. – Anthon Apr 06 '19 at 12:48
  • Hmm yaml seems not the best – Arefe Apr 06 '19 at 12:59
  • can your code read the ‘.yml.j2’ files? I am thinking of using the Jinja2 template for the base of everything – Arefe Apr 06 '19 at 15:01
  • Yes, most jinja2 for YAML can be edited, but you need a plug-in as the jinja2 makes for invalid YAML, Have a look at [this answer](https://stackoverflow.com/a/44515747/1307905) – Anthon Apr 06 '19 at 16:00
1

This is almost definitely inappropriate, but I really can't help myself.

WARNING: This is destructive, and will overwrite .gitlab-ci.yml.

awk '
  NR==FNR && $1=="project_id" {pid=$NF}
  /WE_PROJECT_ID=/ {sub(/\".*\"/, pid)}
  NR!=FNR {print > FILENAME}
' app.py .gitlab-ci.yml

  1. In the first file only, assign the last column to pid only if the first column is exactly "project_id".

  2. On any line in any file that assigns the variable WE_PROJECT_ID, replace the first quoted string with pid.

  3. In any files other than the first, print all records to the current file. This is possible due to awk's nifty buffers. If you have to be told to make a back-up, don't run this.

vintnes
  • 2,014
  • 7
  • 16