Non-greedy regex extraction from JSON with sed

Question

Some JSON:

{"theSecret":"flhiusdh4543sdfasf34sdf+fsfs/sdf43454=","foo":{"bar":41,"something":"hello world","else":"hi","date":20230101,"qux":0,"digest":"sfhusdf234jkkjuiui23kjkj/SFDF34SDSFDS="}}

Prettified:

{
  "theSecret": "flhiusdh4543sdfasf34sdf+fsfs/sdf43454=",
  "foo": {
    "bar": 41,
    "something": "hello world",
    "else": "hi",
    "date": 20230101,
    "qux": 0,
    "digest": "sfhusdf234jkkjuiui23kjkj/SFDF34SDSFDS="
  }
}

I want the value of the theSecret key.

I tried this:

$ echo "$json" | sed -nE 's/.*theSecret":"(.*)".*/\1/p'    # (.*?) doesn't work

Which gives:

flhiusdh4543................4SDSFDS=

i.e. from theSecret until the end of digest. That's because sed lacks non-greedy quantifiers, so .*? doesn't work.

How can I do this?

(This is in an alpine container, which has sed, grep and awk. jq and perl are unavailable.)

Don't use regex to process JSON. Use a tool like `jq` that knows how to parse it. — Barmar, Jul 01 '23 at 04:06
@Barmar I read in other questions that sed lacks the non-greedy quantifier `.*?`. From my test above that seems to be the case. Have I misunderstood, or made a mistake? — lonix, Jul 01 '23 at 04:10
@Barmar "If it lacked ? then it wouldn't match at all." - no. `sed` lacks support of lazy quantifiers, but their presence doesn't break matching: it simple disregards question mark after quantifier, and proceeds with greedy one. Consider this [demo](https://ideone.com/6wljQQ) — markalex, Jul 01 '23 at 07:10
does your grep understand `-P` and `-o`? If so and assuming the value is base64-encoded, `grep -Po '"theSecret":"\K[^"]*' — jhnc, Jul 01 '23 at 15:42
@markalex It's very weird that it accepts them and ignores them, but I agree that it's what the behavior shows. — Barmar, Jul 01 '23 at 19:03
If you feel the duplicate closure of a question isn't correct please edit the question to **explain why the duplicate doesn't apply**. Don't add in comments about the close voter, etc. If you feel there is some problematic pattern with a particular user please **flag for moderator attention**. If you feel there is some discussion needed in the community please take it to [meta] — Abdul Aziz Barkat, Jul 24 '23 at 07:46
If you are editing the question to contest the dupe closure the **only** thing you should add is the explanation of why the target doesn't answer your question (or some clarification / information actually relevant to the question). All of the other commentary you added to your question does not belong there. Again as I mentioned if you want a discussion take it to [meta]. — Abdul Aziz Barkat, Jul 24 '23 at 12:07

score 1 · Accepted Answer · answered Jul 01 '23 at 04:12

1

Instead of using a non-greedy quantifier, use a pattern that doesn't match the terminating ".

$ echo "$json" | sed -nE 's/.*theSecret":"([^"]*)".*/\1/p'    # (.*?) doesn't work

answered Jul 01 '23 at 04:12

Barmar

741,623
53
500
612

1

`[^"]*` matches the longest possible sequence of non-quote characters, which necessarily has to stop before the `"` character. That character is then matched by the `"` after the capture group. – Kaz Jul 01 '23 at 05:44
for this particular case it looks like the value is base64-encoded but, in general, this regex is broken unless you extend it to handle escaped quotes: `'{"theSecret":"mylovely\"secret\""}` eg. `.*theSecret":"(([^\\"]|\\")*)".*` – jhnc Jul 01 '23 at 15:37
Trying to use regular expressions to parse arbitrary JSON is hopeless. It's only feasible when you know that the data is restricted like this. – Barmar Jul 01 '23 at 19:30

Non-greedy regex extraction from JSON with sed

1 Answers1