1

I am trying to call the amazon advertising api and download a campaign report, successfully able to do the following so far:

Create a campaign report, get the report id and using the report id, I am able to get the s3 download url.

The problem is that when I try to download the data - the response returned by the api is supposed to be gunzipped json and comes in the format:

��VJN�-H�L��LQ�2227��45�017�����b%+��܂������< ߤ�+R<

How do I extract the json data from this compressed format? I have searched and found a similar thread on SO:

Why does Amazon Advertising report API return a .bin instead of .json

I am also able to get a valid json file from postman using "Send and Download" option as suggested on this thread.

But stuck when trying to implement the unzip code in nodejs, the code snippet for parsing the response data

await axios.get(downloadurl, options)

        .then(async response => {
         console.log(response.data);
        
        // Calling gzip method
        zlib.gzip(response.data, (err, buffer) => {
  
        // Calling unzip method
        zlib.unzip(buffer, (err, buffer) => {
  
       console.log(buffer.toString('utf8'));
       finaldata =  buffer.toString('utf8');
      
     });
 });

// write data to file
 await fsPromises.writeFile('/Users/asbdb/Downloads/test1/test.json', finaldata);
    
        })
        .catch(error => {
            console.log({ error });
        });

Any pointers on how to unzip and convert the data in string format will be very helpful.

The above code snippet writes this to the output file: output in file

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Please don't copy and paste failed attempts at printing binary data into questions. Convert the binary to hex or base64 and put that in your question. The stream of black diamonds with question marks is completely useless, and prevents people from trying to help you. – Mark Adler Jan 29 '22 at 22:58
  • Thank you Mark for the pointers, I was able to dig more on this and now able to download a .bin file - this is the format being returned by the api. I am new to nodejs and need help in figuring out how the .bin file can be converted/unzipped to a json file. – Jillian Macgowan Jan 30 '22 at 07:40
  • You need to provide the .bin file. Either as a link, or encoded (hex or base64) in your question. If it is very long, put at least the first hundred bytes in the question. – Mark Adler Jan 30 '22 at 17:29
  • Hi Mark, I am attaching the .bin file. I was able to unzip it via an online tool and got the uncompressed file. I am trying to access the data in this file programmatically now - whatever I try, the garbled text comes back from the file. – Jillian Macgowan Jan 30 '22 at 19:57
  • What I have tried so far, 1/ Saved the file to the disk as .json from the api response. 2/ Saved the file to the disk as .bin from the api response. 3/ I have saved the file as ".gz" format also and then tried unzipping it using zlib in nodejs - that's also not working. Attaching the files that were saved to the disk in different formats from the api response. The files are here: https://drive.google.com/drive/folders/1HVE8ddKEmKe9f_vzAI1F9zY8OYl4-LQg?usp=sharing – Jillian Macgowan Jan 30 '22 at 20:10
  • The code snippet that saves the file to the disk – Jillian Macgowan Jan 30 '22 at 20:24
  • await axios.get(downloadurl, options) .then(async response => { console.log(response.data); const w = response.data.pipe(fs.createWriteStream('/Users/jss/Downloads/test1/test.json')); w.on('finish', () => { console.log('Successfully downloaded file!'); }); }) .catch(function (error) { console.log(error); }); – Jillian Macgowan Jan 30 '22 at 20:25

1 Answers1

3

Use zlib.gunzip() to extract the contents.

What you have there is a gzip stream, not a zip file, so zlib.unzip() won't help. zlib.gzip() compresses, as opposed to extracts, You are looking for extraction.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • A good answer will always include an explanation why this would solve the issue, so that the OP and any future readers can learn from it. – Tyler2P Jan 31 '22 at 18:03
  • It would be good to see first if it fixes the problem, but I have added text regardless. – Mark Adler Jan 31 '22 at 18:32
  • Yes this solved the issue - I was able to extract the content from the file, the following snippet worked: – Jillian Macgowan Feb 01 '22 at 15:19
  • This solved the issue! the following snippet worked: 1/ Open the saved file and read data in a buffer 2/ Gunzip using zlib.gunzip() to extract the contents 3/ Extract the contents and print on the console async function readFile() { const buffer = fs.readFileSync("/Users/jss/Downloads/test1/test.json"); console.log("buffer from file: "+buffer); zlib.gunzip(buffer, (err, buffer) => { console.log("buffer data after gunzip extract: "+buffer.toString('utf8')); }); } Data Printed is: [{"campaignId":227589508475555,"clicks":0,"impressions":4}] – Jillian Macgowan Feb 01 '22 at 15:22
  • Thank you so much Mark for helping with this, was stuck for quite some time trying to figure out how to work with this data format – Jillian Macgowan Feb 01 '22 at 15:23
  • Hello, @JillianMacgowan I get incorrect header check error. Did you encounter this error by any chance? – Mj Ebrahimzadeh Feb 19 '23 at 07:49
  • @MjEbrahimzadeh - I did not get this error during the implementation – Jillian Macgowan Mar 15 '23 at 15:24