Perl regex one liner not working correctly

Question

I'm trying to do the following in a bash script to a file:

Before:

{
    "main": "3.3.0",
    "archive":[
        "3.2.2"
    ]
}

After:

{
    "main": "3.3.1",
    "archive":[
        "3.3.0",
        "3.2.2"
    ]
}

using this perl one liner:

perl -pi -e "s/\"main\"\: \"(.*?)(\".*)\[/\"main\": \"${version}\2\[\n\t\t\"\1\",/s" "$json"

Where $version holds the new version number and $json is the path to the file.

It seems perl doesn't match, and I can't understand why. Removing the \[ does match, but I can't see what's wrong with that literal.

@WiktorStribiżew That's a quick closing. This question is about JSON data and not only multiline matching... — oliv, Oct 19 '18 at 13:45
@oliv Added another close reason: [Parsing JSON with Unix tools](https://stackoverflow.com/questions/1955505/parsing-json-with-unix-tools) — Wiktor Stribiżew, Oct 19 '18 at 13:47
@WiktorStribiżew No doubt these are good resource doc to start with, but for sure doesn't answer OP's question about incrementing version number and moving object to a json array... — oliv, Oct 19 '18 at 13:52
Why it doesn't match: it processes the file line by line, and modifiers won't help since the regex simply always has just one line from the file. Then, `main` and `[` are not on the same line; there is no `main.*[` pattern on any lines. You'd need to capture and store the version number for the next lines, or read the whole file into a scalar with `-0777`. But I'd recommend to escape many (easy-error-) details using a module, and to avoid syntax (and maintainability) issues by putting this in a script and run that. — zdim, Oct 19 '18 at 19:07
@zdim thanks for the explanation. For some reason I didn’t consider parsing the JSON - sometimes you get your head too tied up in a problem to see the obvious solution. But an explainer of why this didn’t work will help me understand how Perl works in future. — Nick, Oct 19 '18 at 22:43
@Nick You are welcome, let me know if more can help. Getting drawn in is completely understood. There is a lot that one _can_ do with a regex -- but it also gets tricky very quick, and it is often a blessing when you can do it nicely otherwise; a similar (and stronger) statement goes for a "one-liner." As soon as you feel like you're cramming more and more into it (a "one-liner" that almost invariably comes with a nasty regex) to make it work it's likely time to step back, open up a file for the script, and reconsider. — zdim, Oct 19 '18 at 23:07

zdim · Accepted Answer · 2018-10-21T05:18:55.533

Assuming that you can fix your JSON (a trailing comma is not allowed) I'd recommend to process this using a module, and in a script.

An example with JSON, wrapped in a "one"-liner as asked

perl -MPath::Tiny -MJSON -0777 -wnE'
    my $hr = decode_json $_; 
    unshift @{$hr->{archive}}, $hr->{main}; 
    $hr->{main} =~ s/[0-9]+\.[0-9]+.\K([0-9]+)/$1+1/e; 
    path("new_".$ARGV)->spew(encode_json $hr)' 
' data.json

The very handy Path::Tiny is used to easily dump JSON output. I make the output file name by prefixing new_ to the input filename (available in $ARGV variable), adjust as suitable.

If installing a module is a problem for some reason you can instead simply print the JSON encoded string and redirect the output

perl -MJSON -0777 -wnE'
    my $hr = decode_json $_; 
    unshift @{$hr->{archive}}, $hr->{main}; 
    $hr->{main} =~ s/[0-9]+\.[0-9]+.\K([0-9]+)/$1+1/e; 
    say encode_json $hr 
' data.json > new_data.json

These produce the output file with &hairsp; {"archive":["3.3.0","3.2.2"],"main":"3.3.1"}

With the -0777 command switch the whole file is "slurped" into a scalar ($_) and -M... loads the given module. Then we use decode_json, which JSON exports by default in its functional interface, to obtain a hashref with data.

The current value of main is then added to the beginning of the arrayref in archive using unshift, and is then changed by bumping up its release/patch number, using a regex.

Finally the encode_json, also exported in JSON's functional interface, is used to JSON-encode the hashref, what is dumped either with Path::Tiny to a file or to STDOUT for redirection.

A word on the number of existing JSON modules is in order, prompted by comment by Grinnz.

The JSON linked above will load the JSON::XS module, and if that is not installed then it falls back to the compatible pure-Perl one, JSON::PP. I'd recommend the XS module, which is much faster and proven in wide use.

Another option is Cpanel::JSON::XS, the JSON::XS fork with a bug tracker and a list of bug fixes. I've happily used both without any issues.

To have these tried in order Cpanel:: then ::XS then ::PP, also with some improvements over JSON in how backends are loaded, use JSON::MaybeXS.

Note that none of these are in core.

May I suggest recommending JSON::MaybeXS instead of JSON; neither is core, but JSON does not default to the currently-recommended Cpanel::JSON::XS backend, and adds significant overhead to any backend it uses, whereas JSON::MaybeXS does not have these issues and is largely a drop-in replacement. — Grinnz, Oct 20 '18 at 17:52
@Grinnz Added a comment, thanks for prompting it I am aware of talk of the missing bug tracker and lack of responsiveness (for ::XS), but I've seen counter-claims as well and without a firm knowledge of the matter (and no first-hand experience) I can't directly recommend to go this way or that. Thus a generic comment and people can research and make choices. — zdim, Oct 20 '18 at 19:55
What's up with all the -1's on this page? Nothing posted is any good? — zdim, Oct 22 '18 at 17:42

score 1 · Answer 2 · answered Oct 19 '18 at 15:56

(now that Ihis question has been reopened and I can post an answer..)

Assuming you fix your data so it's valid JSON (Note the invalid trailing comma), this is an easy jq one-liner:

$ jq "{main: \"$version\", archive: [ .main, .archive[] ]}" "$json" > new.json
$ mv -f new.json "$json"

Just like with HTML and XML, using regular expressions to try to manipulate JSON is a mistake. Use a more appropriate tool.

score 1 · Answer 3 · answered Oct 31 '18 at 12:53

Using jq to both move the main version number to the array archive and increment the last digit of the main version:

jq '.archive += [.main] |
    .main |= (split(".") | .[-1] |= (tonumber+1|tostring) | join("."))' file

The operator += adds a new value to the array archive.

The operator |= assigns a new value to main. The last element of the splitted string is incremented by 1 and then reassembled.

Perl regex one liner not working correctly

3 Answers3