iterate through yaml using shell

Question

I have a config_file.yml file as:

sample:
    sql: "select * from dbname.tableName where sampleDate>='2018-07-20';"
    config: {'hosts': [!!python/tuple ['192.162.0.10', 3001]]}

sample2:
    sql: "select * from dbname.tableName where sampleDate<='2016-05-25';"
    config: {'hosts': [!!python/tuple ['190.160.0.10', 3002]]}

I want to iterate through its key value pairs till EOF using shell script. Basically I want to be able to iterate through each sql till EOF, and execute each sql in a shell loop.

Tried looking through a lot of docs but they dnt have enough info how to loop through yaml using shell.

Any ideas or example will be very helpful...

Thanks!

EDIT:

I am already using ->

parse_yaml() {
local prefix=$2
local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
sed -ne "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" $1 |
awk -F$fs '{
  indent = length($1)/2;
  vname[indent] = $2;
  for (i in vname) {
     if (i > indent) {delete vname[i]}}
  if (length($3) > 0) {
     vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
     printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
  }
}'
}
# read yaml file
eval $(parse_yaml config_file.yml "")
# access yaml content
echo $sample_sql

I am not able to understand how I can iterate through this.

Just write your own simple bash parser. https://stackoverflow.com/questions/5014632/how-can-i-parse-a-yaml-file-from-a-linux-shell-script?rq=1 — KamilCuk, Jul 22 '18 at 20:10

moebius · Accepted Answer · 2018-07-23T20:32:09.440

4

Any reason why you can't use python?

It should come pre-installed with most linux distributions, and it means you don't have to re-invent the wheel!

To setup:

pip install pyyaml

Then your script is:

import yaml
f = open('config_file.yml')
yaml_file = yaml.safe_load(f)
for sample in yaml_file:
    print yaml_file[sample]["sql"]

To run:

python <script_name>.py

edited Jul 23 '18 at 20:32

answered Jul 23 '18 at 12:14

moebius

2,061
11
20

2

As specified in the PyYAML documentation `yaml.load()` can be unsafe. There is no excuse for using that, instead of `yaml.safe_load()`. You should also not polute your system's Python installation with `pip` installs, always create a virtualenv for your utiltities, so `pip` installs cannot break your system. – Anthon Jul 23 '18 at 13:27
Yes i finally switched to python... this works perfectly – user3868051 Jul 23 '18 at 15:51

score 0 · Answer 2 · answered Jul 23 '18 at 12:44

You could use grep or pcregrep with something like this:

$ while read -r line; do
  echo $line
done < <(grep -oP "sql: \"\K.+?(?=\")" config_file.yml)

Output:

select * from dbname.tableName where sampleDate>='2018-07-20';
select * from dbname.tableName where sampleDate<='2016-05-25';

In macOS, you could use pcregrep -o "sql: \"\K.+?(?=\")" config_file.yml

The \K can be read as excluding everything to the left before it and return only the right part .+?(?=\") until " is found.

Now if your yaml is more complex you could give a try too yq (pip install yq) and then, for example, to get the sql values you could do:

$ yq -r '.sample.sql' config_file.yml

SwiftNinjaPro · Answer 3 · 2023-06-19T19:22:32.127

I modified the script a bit to include the keys in a space separated list.

note: this method requires a prefix.

function parse_yaml {
  local prefix=$2
  local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
  sed -ne "s|^\($s\):|\1|" \
    -e "s|^\($s\)\($w\)$s:$s[\"']\(.*\)[\"']$s\$|\1$fs\2$fs\3|p" \
    -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p"  $1 |
  awk -F$fs '{
    indent = length($1)/2;
    vname[indent] = $2;
    for (i in vname) {if (i > indent) {delete vname[i]}}
    if (length($3) > 0) {
      vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")}
      printf("%s%s%s=\"%s\"\n", "'$prefix'",vn, $2, $3);
      printf("%1$s%2$s=\"${%1$s%2$s} %3$s\"\n", "'$prefix'",vn, $2, $3);
      printf("%1$s=\"${%1$s} %2$s\"\n", "'$prefix'",vn, $2, $3);
    }
  }'
}

usage

eval $(parse_yaml "config_file.yml" "CONF_")

# to specify all keys (apart from the final level)
for key in $CONF_; do
  echo "$key"
done

# or to list keys of a specific object
for key in $CONF_sample_; do
  echo "$key"
done

Note: This method is limited to the core level, and final level. Someone else may be able to modify this to use in between levels if they need that feature.

iterate through yaml using shell

3 Answers3