in bash, create json object of key=filename and value=file-contents given sequence of pathed filenames on stdin

Question

For example, if the list of the filenames on stdin is /etc/alpha.txt and /tmp/beta.txt

And /etc/alpha.txt contains wibble

And /tmp/beta.txt contains fu\nbar

Then what I'd like to generate is

{"/etc/alpha.txt":"wibble","/tmp/beta.txt":"fu\nbar"}

I don't have access to any programming languages. This is on a Linux OS. I can install utilities like jq.

The solution from Léa Gris looks spot on. Thank you Léa. Alas my question has been closed as not being focused enough. Sorry about that. This is only my second question on StackOverflow! I'm struggling to make it more focused. This really is my exact issue. I'm trying to make the core runner service in https://cyber-dojo.org a little faster.

My attempts had got stuck at what to put before the jq -s add.

Several options at: https://stackoverflow.com/questions/38860529/create-json-using-jq-from-pipe-separated-keys-and-values-in-bash — hesham_EE, May 14 '20 at 21:39
To avoid close and down-votes, please show at least one of the attempts you've made. — peak, May 14 '20 at 22:08

Léa Gris · Accepted Answer · 2020-05-15T07:01:13.033

Here it is:

#!/usr/bin/env bash

files=('alpha.txt' 'beta.txt' 'delta with space name.txt')

# Filling sample files
printf 'wibble' >'alpha.txt'
printf 'fu\nbar' >'beta.txt'
cat >'delta with space name.txt' <<'EOF'
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
EOF

# Stream the test files names (1 per line)
printf '%s\n' "${files[@]}" |
  xargs -L 1 -I {} jq -sR --arg key {} '{ ($key): .}' {} | jq -s 'add'

xargs -L 1 -I {}: Executes a command by passing each line from stdin while substituting curly braces by the file name.

xargs then runs the jq command to create the filename key and value content of file objects:

jq -sR --arg key {} '{ ($key): .}' {}

Finally, the stream of JSON objects is piped into a final jq -s 'add' to re-assemble it, into a merged object with "key": "value" pairs:

jq -s 'add'

And finally the actual output of all this:

{
  "alpha.txt": "wibble",
  "beta.txt": "fu\nbar",
  "delta with space name.txt": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\nUt enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.\nDuis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.\nExcepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n"
}

Processing all files with a single `jq`:

xargs -I {} jq -sR '{(input_filename):.}' {} | jq -s add

The caution about using input_filename from man jq:

input_filename

Returns the name of the file whose input is currently being filtered. Note that this will not work well unless jq is running in a UTF-8 locale.

This is a very elegant solution! One small question - the resulting JSON has trailing newlines for both keys and values. Can they be stripped to match OP's question? — bgstech, May 14 '20 at 23:59
@bgstech I see no trailing newlines in file names keys in my JSON output. Maybe something about specific `jq` version or running environment. — Léa Gris, May 15 '20 at 00:29
@bgstech greatly simplified the process by removing the need of an intermediate shell script, but using a substitution string to provide arguments to `jq`. Also improved the final `jq` that can merge any number of objects from the input stream. — Léa Gris, May 15 '20 at 00:49

score 0 · Answer 2 · answered May 14 '20 at 21:36

Here's one option that assumes both jq and a bash or bash-like shell:

while read -r d
do
  (cd "$d"; find . -type f -maxdepth 1) | while read -r f ; do
      echo $d $'\t' $(sed 's,^./,,' <<< "$f")
  done
done | jq -Rn '[inputs | split("\t") | {(.[0]): .[1]}] | add'

score 0 · Answer 3 · answered May 14 '20 at 22:00

0

For reference, if you don't want to use jq:

#!/bin/bash

gulp() {
    echo $(sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' $1)
}

while read f1 && read f2; do
    key=$(gulp $f1)
    val=$(gulp $f2)
    echo "{\"$f1\":\"$key\",\"$f2\":\"$val\"}"
done

The "gulp" subroutine reads a file and converts newlines to literal '\n'. Then the main part of the script just reads two lines at a time from stdin.

answered May 14 '20 at 22:00

bgstech

624
6
12

Body of "gulp" based on: https://stackoverflow.com/questions/38672680/replace-newlines-with-literal-n – bgstech May 14 '20 at 22:01

in bash, create json object of key=filename and value=file-contents given sequence of pathed filenames on stdin

3 Answers3

Processing all files with a single `jq`:

Linked

in bash, create json object of key=filename and value=file-contents given sequence of pathed filenames on stdin

3 Answers3

Processing all files with a single jq:

Linked

Processing all files with a single `jq`: