Bash script for TXT file to JSON conversion parsing issue

Question

I have to create a simple bash script that converts .txt outputs to JSON code. The pattern is as follows:

[ Example ], 1..3 tests
-----------------------------------------------------------------------------------
not ok  1  text1 (text1), 5ms
ok  2  text2 (text2, issues), 2ms
ok  3  text3 (text3, issues), 15ms
-----------------------------------------------------------------------------------
2 (of 3) tests passed, 1 tests failed, rated as 66.66%, spent 22ms

This will have to be converted to the following JSON (as an example):

{
 "testName": " Example ",
 "tests": [
   {
     "name": "",
     "status": false,
     "duration": ""
   },
   {
     "name": "",
     "status": true,
     "duration": ""
   }
 ],
 "summary": {
   "success": ,
   "failed": ,
   "rating": ,
   "duration": ""
 }
}

I managed to almost complete the requirements, the test name is taken out correctly from the txt and also the summary results are taken correctly, but the only problem is with the tests part which does not work at all, not even the variables set upfor that are recognosed.

My code is:

#!/bin/bash

###################PREREQUISITES#####################

# Check if the output file is provided as an argument (I chose to do so to be able tospecify location)

if [ $# -eq 0 ]; then
  echo "Usage: $0 <output_json_file>"
  exit 1
fi

input_file="example_file.txt"  #File that is being used as source to convert FROM
output_file=$1    #Output file location as an argument to convert TO

# Check if the input file exists

if [ ! -f "$input_file" ]; then
  echo "Input file '$input_file' not found."
  exit 1
fi

###################CODE#####################

#Setting up a pattern variable for the lines with the values that we need as JSON variables:

pattern='"^(ok|not ok)[[:space:]]{2,}([0-9])[[:space:]]{2,}(.+),[[:space:]]+([0-9]{1,3}ms)$"'

# Read the input_file line by line:

while IFS= read -r line; do
  # Extract the test_Name from the first line -> IT WORKS
  if [[ "$line" =~ \[([^\]]+)\] ]]; then
    testName="${BASH_REMATCH[1]}"
  # Extract values using BASH_REMATCH on the pattern variable -> IT DOES NOT WORK
  elif [[ "$line" =~ $pattern ]]; then
      status="${BASH_REMATCH[1]}"
      status_code="${BASH_REMATCH[2]}"
      name="${BASH_REMATCH[3]}"
      duration="${BASH_REMATCH[4]}"
    break
  elif [[ "$line" =~ ^([0-9]+)\ \(of\ [0-9]+\)\ tests\ passed,\ ([0-9]+)\ tests\ failed,\ rated\ as\ ([0-9.]+)%,\ spent\ ([0-9]+)ms$ ]]; then
      success="${BASH_REMATCH[1]}"
      failed="${BASH_REMATCH[2]}"
      rating="${BASH_REMATCH[3]}"
      duration="${BASH_REMATCH[4]}ms"
    break  
  fi
done < $input_file

# Construct the JSON output
output_json="{
 \"testName\": \"$testName\",
 \"tests\": [
   ${tests[@]}
 ],
 \"summary\": {
   \"success\": $success,
   \"failed\": $failed,
   \"rating\": $rating,
   \"duration\": \"$duration\"
 }
}"

# Write the JSON output to the output file
echo "$output_json" > "$output_file"

echo "Conversion completed. JSON output written to $output_file"

As for now the status is that either only the Name works, so the TESTS and SUMMARY section is not filled in to the JSON file, or (if modified with removing tests) the NAME and SUMMARY works but TESTS dont.

What am I doing wrong? Is there something I am doing fundamentally wrong with this?

Thanks for the advice in advance!

Remove the double-quotes from the `pattern`, they don't belong there. The final result refers to a `test` array that hasn't been assigned at all; it looks like the `[[ "$line" =~ $pattern ]]` section should add to that array, and *not* `break`. Also, expanding a bash array inside a string will not produce a valid JSON array. For this and many similar reasons, you shouldn't use bash to create JSON directly, you should use something like `jq` to JSONify the data (see [this](https://stackoverflow.com/questions/48470049) and [this](https://stackoverflow.com/questions/22434290)). — Gordon Davisson, Sep 01 '23 at 09:07

score 2 · Answer 1 · answered Sep 01 '23 at 12:51

jq as mentioned is the "go-to" tool for dealing with JSON.

One useful feature is the capture() function which will build an object from named capture groups.

name_re='^\[(?<testName>[^]]+)\]'

test_re='(?<status>^(?:not )?ok) +(?<status_code>[0-9]+)'
test_re+='+(?<name>.+), +(?<duration>[0-9]+)ms'

summary_re='^(?<success>[0-9]+) \(of [0-9]+\) tests passed, '
summary_re+='(?<failed>[0-9]+) tests failed, rated as (?<rating>[0-9.]+)%'
summary_re+=', spent (?<duration>[0-9]+)ms$'

jq --arg name_re    "$name_re"    \
   --arg test_re    "$test_re"    \
   --arg summary_re "$summary_re" \
   -n -R '

[inputs] as $lines |
($lines[0] | capture($name_re)) + {
  "tests": [$lines[] | capture($test_re)], 
  "summary": $lines[-1] | capture($summary_re)
}

' input.txt

{
  "testName": " Example ",
  "tests": [
    {
      "status": "not ok",
      "status_code": "1",
      "name": "  text1 (text1)",
      "duration": "5"
    },
    {
      "status": "ok",
      "status_code": "2",
      "name": "  text2 (text2, issues)",
      "duration": "2"
    },
    {
      "status": "ok",
      "status_code": "3",
      "name": "  text3 (text3, issues)",
      "duration": "15"
    }
  ],
  "summary": {
    "success": "2",
    "failed": "1",
    "rating": "66.66",
    "duration": "22"
  }
}

score 1 · Answer 2 · answered Sep 01 '23 at 20:22

Solution in TXR:

$ txr log2j.txr testlog
{
  "tests" : [
    {
      "name" : "text1",
      "status" : false,
      "duration" : "5"
    },
    {
      "name" : "text2",
      "status" : true,
      "duration" : "2"
    },
    {
      "name" : "text3",
      "status" : true,
      "duration" : "15"
    }
  ],
  "testName" : "Example",
  "summary" : {
    "success" : "2",
    "failed" : "1",
    "duration" : "22",
    "rating" : "66.66"
  }
}

If some of the numeric values should be numbers, which the question doesn't indicate, some adjustments can be made.

Code in log2j.txr:

[ @testname ], @nil tests
-----------------------------------------------------------------------------------
@(collect)
@  (cases)
not ok @num @name (@name), @{dur}ms
@    (bind status nil)
@  (or)
ok @num @name (@name, @nil), @{dur}ms
@    (bind status t)
@  (end)
@(last)
-----------------------------------------------------------------------------------
@(end)
@succ (of @total) tests passed, @fail tests failed, rated as @percent%, spent @{totaldur}ms
@(do (let ((*print-json-format* :standard)
           (tests (vec-list (mapcar (ret #J^{ "name" : ~@1,
                                              "status" : ~@2,
                                              "duration" : ~@3 })
                                    name status dur))))

       (put-jsonl
         #J^{ "testName" : ~testname,
              "tests" : ~tests,
              "summary" : { "success" : ~succ,
                            "failed" : ~fail,
                            "rating" : ~percent,
                            "duration" : ~totaldur }})))

glenn jackman · Answer 3 · 2023-09-01T12:35:41.757

2 observations:

you're not actually building the tests array that you use to construct the JSON output
use jq to build JSON

#!/usr/bin/env bash

{
    IFS= read -r header
    if [[ $header =~ "["([^]]+)"]," ]]; then
        testName=${BASH_REMATCH[1]}
    fi
    read
    tests=()
    passed=0 failed=0
    while IFS= read -r line; do
        [[ $line =~ ^-+$ ]] && break
        if [[ $line =~ ("not ok"|"ok")"  "[0-9]+"  "(.*)", "(.*s)$ ]]; then
            if [[ ${BASH_REMATCH[1]} == "ok" ]]; then
                status=true
                ((++passed))
            else
                status=false
                ((++failed))
            fi
            test=$(jq -nc --arg name "${BASH_REMATCH[2]}" \
                          --argjson status "$status" \
                          --arg duration "${BASH_REMATCH[3]}" \
                      '$ARGS.named')
            tests+=("$test")
        fi
    done
    IFS= read -r footer
    if [[ $footer =~ "rated as "(.*)", spent "(.*) ]]; then
        summary=$(jq -nc --argjson success "$passed" \
                         --argjson failed "$failed" \
                         --arg rating "${BASH_REMATCH[1]}" \
                         --arg duration "${BASH_REMATCH[2]}" \
                     '$ARGS.named')
    fi

    IFS=","
    jq -n --arg testName "$testName" \
          --argjson tests "[${tests[*]}]" \
          --argjson summary "$summary" \
       '$ARGS.named'

} < test_results.txt

Bash script for TXT file to JSON conversion parsing issue

3 Answers3