0

Sample YAML file:

key1: value1
key2: value2
key3: value3

How can I add the keys key2a and key2b after key2 using Mike Farah’s yq? Although the order of keys is technical and of minor importance, YAML files frequently serve as configuration files. I want to maintain a specific order and grouping for human editors.

Desired result:

key1: value1
key2: value2
key2a: value2a
key2b: value2b
key3: value3

A similar question was asked some time ago but referred to the Python version of yq.

Stephan
  • 488
  • 2
  • 4
  • 15

1 Answers1

1

Using mikefarah/yq, you could traverse and rebuild the structure using ireduce, while inserting more items if conditions match.

Note #1: This approach uses the alternative operator // to compensate for an if conditional which is still missing in yq's most recent release v.4.31.1 (Feb 20, 2023).

(.[] | [key, .]) as $i ireduce({}; .[$i[0]] = $i[1]
  | select($i[0] != "key2") // (
    .["key2a"] = "value2a" | .["key2b"] = "value2b"
  )
)
key1: value1
key2: value2
key2a: value2a
key2b: value2b
key3: value3

However, until v4.27.5 (Sep 9, 2022) there was another bug in yq that wrongly evaluated assignments in the alternative branch (RHS of //) which eventually breaks this approach.

Thus, for prior versions of yq, you could either replace the assignments with the adding of pre-formatted objects, or replace the whole alternative expression with the context operator with by abusing its root context for the condition:

# Workaround for yq < v4.27.5, using object addition

(.[] | [key, .]) as $i ireduce({}; .[$i[0]] = $i[1]
  | select($i[0] != "key2") // (
    . + {"key2a": "value2a", "key2b": "value2b"}
  )
)
# Workaround for yq < v4.27.5, using the `with` operator

(.[] | [key, .]) as $i ireduce({}; .[$i[0]] = $i[1]
  | with(select($i[0] == "key2");
    .["key2a"] = "value2a" | .["key2b"] = "value2b"
  )
)

Note #2: All of these approaches also use the key operator, which was only introduced in v4.15.1 (Nov 24, 2021). Thus, for even prior versions of yq, you'd need other workarounds...


With kislyuk/yq you do have the if expression available, but you can likewise just take the select shortcut. Also, you can simply return a comma-separated stream inside a map, which is internally called by with_entries.

# Using `if`

with_entries(if .key == "key2" then .,
  {key: "key2a", value: "value2a"},
  {key: "key2b", value: "value2b"}
else . end)
# Using `select`

with_entries(., (select(.key == "key2") |
  {key: "key2a", value: "value2a"},
  {key: "key2b", value: "value2b"}
))

With itchyny/gojq you'd probably have a hard time as it automatically sorts object keys by design. Thus, if your input file has keys in an unsorted order (e.g. key3, key1, key2), then even processing just the identity filter . would yield an output with its keys sorted (e.g. key1, key2, key3). Evidently, the same goes for any newly inserted keys.

However, if your actual keys (including the ones to be inserted) are by chance in a sorted order anyway (like in your sample input, literally), you can exploit this circumstance by just primitively appending the new items at the end, and gojq will take care of inserting them at their sorted positions.

pmf
  • 24,478
  • 2
  • 22
  • 31
  • Thanks for your yq expression. Unfortunately, the logic fails when the keys in the sample input file have a different order (e.g., `key3`, `key1`, `key2`). The expression then adds `key2a` and `key2b` after `key3` instead of `key2`. – Stephan Feb 28 '23 at 17:55
  • @Stephan I fail at replicating your observation. Moving the third line to the top in the input file, while executing the same snippet (i.e. still referencing `key2` which is then found last in the input file) yield the expected result (in the following order `key3`, `key1`, `key2`, `key2a`, `key2b`). Tested with yq 4.30.5 -- Edit: Your described behavior was indeed replicable with an older version of yq (namely 4.20.2), so updating might solve the issue. – pmf Feb 28 '23 at 18:28
  • I highly appreciate your feedback. The yq version is a good point (currently on v4.27.3). I will update and re-test. – Stephan Feb 28 '23 at 18:39
  • @Stephan While skimming through the [change log](https://github.com/mikefarah/yq/releases) I found a fix released with [v4.27.5](https://github.com/mikefarah/yq/releases/tag/v4.27.5) reading "Fixed bug in alternative (//) operator, RHS being evaluated when it didn't need to be" which could reasonably be the cause. – pmf Feb 28 '23 at 19:49
  • @Stephan See my update with two new approaches avoiding that bug, lowering applicability to [v4.15.1](https://github.com/mikefarah/yq/releases/tag/v4.15.1)+. – pmf Mar 01 '23 at 04:19
  • Thank you so much for your efforts and comprehensive answer to my question. Your solution indeed works well. I’ve decided to go forward and update to the latest yq release. – Stephan Mar 01 '23 at 18:32