Sed variable too long

Question

I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}

sed -i "s|{FILES}|$FILES|" ./myFile.json

What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :

FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."

I can't think of a solution where xargs would help me.

It would be useful if you could provide us with a reduced, representative sample of your input and desired output. — Tom Fenech, Mar 23 '15 at 16:13
And you want the whole list wherever {FILES} was found in your json file? — Peter Bowers, Mar 23 '15 at 16:18
Question edited, added a sample, and {FILES} is unique, so the whole list replacing that string. — Bastien974, Mar 23 '15 at 16:20
Would the answer to http://stackoverflow.com/questions/28865473/setting-environment-variable-to-a-large-value-argument-list-too-long help you here? If you can put your aws command in the same script without the export? — Peter Bowers, Mar 23 '15 at 19:26

score 2 · Accepted Answer · edited May 23 '17 at 12:22

The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.

For example, if myFile.json would contain:

{foo: 1, bar: "{FILES}"}

then the script should be:

#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF

You can generate the script with a single sed command:

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json

Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.

Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash

Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin

Output:

{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}

Very nice trick. Is there another way to call bash as a subshell, so that $FILES doesn't need to be exported as env variable ? — Bastien974, Mar 25 '15 at 13:45
@Bastien974 Just replace `/bin/bash` by `source /dev/stdin`; see my edit. — Ruud Helderman, Mar 25 '15 at 14:45

score 0 · Answer 2 · answered Mar 23 '15 at 16:15

0

Maybe perl would have fewer limitations?

perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json

answered Mar 23 '15 at 16:15

Peter Bowers

3,063
1
10
18

Tried, same limitation. – Bastien974 Mar 23 '15 at 16:21
I'm really surprised. Perl and sed both have pretty amazing buffer limitations (or lack thereof)... – Peter Bowers Mar 23 '15 at 16:52
perl: Argument list too long – Bastien974 Mar 23 '15 at 17:27

score 0 · Answer 3 · answered Mar 23 '15 at 16:25

0

It's a little gross, but you can do it all within shell...

while read l
do
    if ! echo "$l" | grep -q '{DATA}'
    then
        echo "$l"
    else
        echo "$l" | sed 's/{DATA}.*$//'
        echo "$FILES"
        echo "$l" | sed 's/^.*{DATA}//'
    fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json

Obviously I'd leave the final line commented until you were confident it worked...

answered Mar 23 '15 at 16:25

Peter Bowers

3,063
1
10
18

Yeah it works, i'll use that if there's no better alternative. – Bastien974 Mar 23 '15 at 17:51

score 0 · Answer 4 · answered Mar 23 '15 at 17:41

0

Maybe just don't do it? Can you just :

echo "var f = " > myFile2.json
echo $FILES >> myFile2.json

And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)

answered Mar 23 '15 at 17:41

Barett

5,826
6
51
55

The json file is used to feed another command: "aws cloudfront create-invalidation --distribution-id xxx --invalidation-batch "$(< ./myFile.json)"), so I'm not able to reference. – Bastien974 Mar 23 '15 at 17:48

score 0 · Answer 5 · answered Mar 23 '15 at 19:54

Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:

foo.pl:

open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
   s/world/$foo/;
   print;
}

Command to run:

aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json

Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.

Sed variable too long

5 Answers5