Fetch value from column name in unix

Question

val=$(curl https://www.geeksforgeeks.org | tr '\n' ' ' | tr -s [[:space::]] | grep -E client_id -e publication_id)

I have a requirement to fetch the particular value from curl command:

Sample I/P:
client_id:"15647",publication_id:["151","152","153"]

Output:

publication_id:["151,"152","153"]

I want to use grep command to fetch publication_id values from curl command in unix and store in variable

And this is not faisable with Curl, as far as the HTML matching `client_id` is JS generated — Gilles Quénot, Feb 16 '23 at 07:45
we cannot install anything in vdi because its restricted.. Can you provide alternative solution — shikha. agarwal, Feb 16 '23 at 07:49
Please take care of my comment, **this is generated by JS**. NoGo. Need https://pptr.dev or Selenium (javascript aware library) — Gilles Quénot, Feb 16 '23 at 08:12
I wanted to do by unix..If there is any approach can you please tell me — shikha. agarwal, Feb 16 '23 at 08:28
See: [Parsing JSON with Unix tools](https://stackoverflow.com/q/1955505/3776858) — Cyrus, Feb 16 '23 at 08:37
**it"s not possible to parse this JSON with unix tools as far as it's generated by javascript** — Gilles Quénot, Feb 16 '23 at 08:43

Gilles Quénot · Answer 1 · 2023-02-16T15:05:31.297

One solution with nodejs, xidel, xpath and puppeteer, as said in comments, you need a js aware tool:

Javascript wget.js code (this is only to mimic wget, it's output only HTML, and the one generated by js too):

const puppeteer = require('puppeteer');

(async () => {
    var url = process.argv[2];
    const browser = await puppeteer.launch({headless: true});
    const page = await browser.newPage();
    await page.goto(url, { waitUntil: 'networkidle2' });
    const html = await page.evaluate(() => document.documentElement.outerHTML);
    console.log(html);
    browser.close();
})()

code:

node wget.js 'https://www.geeksforgeeks.org' |
    xidel -e '//div[@id="g_id_onload"]/@data-client_id'

Output:

388036620207-3uolk1hv6ta7p3r9l6s3bobifh086qe1.apps.googleusercontent.com

Fetch value from column name in unix

1 Answers1