How to add headers in node.js

Question

I have the following headers:

X-NewRelic-ID: VQACUF9bCBABU1JQAAQPU1I=
X-Device-Descriptor-Id: YWY1Zjk5OGI5N2U1ODg0Zg
User-Agent: Sahibinden-Android/4.5.2 (671; Android 5.1.1; Android Android SDK built for x86)
x-api-key: e91092ad5ea2e030c201ce9ac4373f6b565a7842
x-client-profile: Generic_v2.1
x-device-id: d4dc022224a74bc7964cb4d449cd32c2
x-timestamp: 1591117620862
x-api-hash: DF7AB8944EDB3140BCB2A3EA9BC95F70C3200DB7
adid: b165e39f-b0ee-4b22-aee5-df3675136c2c
Content-Type: application/json; charset=utf-8
X-J3PopQvX-e: a;grA22x5m7f3tr-0iQLVuq4BLsOW3uMVgYrreUpRM5M9sSufn0I0K5QO-LrGIm1gY8IHM_GtH_7uQ-cRTU-eV9762ivhZAOpU0I2ZC_dii-H-LDjjBUj8dI1uMmlx8sJjURVN1NWhJuf7iYYkopysE4rI5Ku0OAq-vv8po6yl7vF6KA4c1JEoyeJHX0WImy3c4SV30gzj4fLe1HfYlsA1Qa3ea7pl6kE4fXnYZ0q9DbWcxT_xUBj2Hx7GPYqqdOy8LK2eVQtj_qIyBqlUMu_B1WBteBo8JpEDsCN6hVZVVOmCvzBFVrJgh3rXd-bz3J6-TAgFOgIlbKEob6SrxPFvmrjrjFz2CESgBYUe2YBkwzOMYiNfRLfYKRF9bifV53JcX25D6ycfE5wp72kRvUDsF049QUgUyEvAOIdZcOTbETQtoAOIYtvClrsZQ8lI8VuKpHaqm3z0Cz59jNvXDYSIyVGQ43BbFHYJmvuVDK5hms8YQe1AU4z-kWFEZbR1GI2SD3e1ivl5a8YQNj3YLId6HeHy1BZZ4vrMfKhRmW6RFz3E8--TDu3r9J_RyYOh7_UhJ6-KW1FQ4mnoAD-1ObhBgw7VvefrijzY6iQKbnVhR2GVs0eAMv8RvON81vgIE7eZvqv4TFbNfpFzbNDYE4JboRsrroyElH9kbRkh6xc8Y_dlXZnZqHZH8Z55f1rBAQjeci9YzxGUJ4ssJE2vlniRkUSf1IZEwpxDdeUmKjk95EmdNIuC3NapbQbeRumyoe7XBCN3GOJ860o1wfeVE_u6aPpwtWAU7Huc2wY4-0M8V-WraljaUYt3iH2qVV9K-GryALZujOmjl6lLZTFBEFqijc8=;oh90F-L4vdHrRPBFFSIfspu1OBIaoliYHzKnk2hKNKM=
X-J3PopQvX-a: 0JMyqk3jVhB_ktkVhvJjiwUXW1jwIGUwnf4AuOufJI0WLr3Urn4m1ANxyELGawEfHAh3vvz7CcPSjuByQZVxMm-jqYz7WjSseSmlZGLnkSqTaTTYN7Rfz7aZTf2fSHc-F5zaQdRGr3t_KDw17Wu1CTjo8QGWUizaHYjkkclkI2q9R7McZmd_K2dpbBGOFwuVeuH4BoTabg7m9PxQ5PoqFykgD2dp=1a8_-iWK85xHc7N0iC7Kr0CmVECcipNabR42f=D15vshVt0cxZaZZd9hFHoY_b9RrDR5AjRvUTznXmJalncdoR0Uf9oP8DEv=uwgpWYp2O1szeXLHhrH9x9uutcDvyvbikmfl4G3qMa1b9TFvkWQCU7xEKN0qE5MTF2JOkJP0K-tRspZTm1dV7hFISNrU-nNJnDr0NI7l8-97eot5jkcTL=I=_7NDicD=8LlPbyWHeWBts3RJBQndMnW9p=J2Kth5rDIrHzfxsodpT0PsEHYY--EGX7-ijux_Xy4BTzYfYIc1g5teDskpD11nxM1Z=ioeOo0Ig-ocuf9TlHQPZcGru8koECDU8AKu_eptN_B50=Molip5MRHnD994ZKDjzv3N2I85rGzEZIjAT33_VSKNcqdsPSkDzUNwYolSc272M3=mAMtA2j3vWPoTuBdxkNZ3BIlBiE2k5sjiAG8upeDiVhAOzQB8BeO2LmP4VHH88Ng5brkoHeX3ncLD4WkIw0R-9TAou3zNmsClfJ_0dszt24ISM-H0GNDwuKKFm92JswKvWQOrrRGAYZODRJCzQAZQgjn=oxnjxiznUegsD84foGvwAxZyZZq8yMGNuEUk0pePeQjxWmZznzAAolQer1JvM0iHGjRJwwfXnzTF71H8GElk-sPFYDOzje11ozUnOwgom-o5FYemTpj8aR4uHffTrQa1KwXK_KMkeuJY0Q2mXNXJo49CSbO=lqWL1fNxi5OTZsYU7q2jXCSStaNR1DuTymMBJO3IFBeNFZDysgaSFooy_4H8Oi4G-eXNkuTE3sT-m-ZRleF8aVbTuhWo_ZgWuQDHAgELj2wCeGHnmh1_AeewO4wM55Le-YSC2hGHtg2qXoruUCvlQ9XJ=pcS7PLEMfEGbpbxvOaPnHAxiMc1Ahq54ZIzo-tuxDBlrrBO9FMq0a__z_CJxD0hSgWBaF2nnrrKS4ijOOhP8s1T7nPEdEDQcbVO9Q_wNd8CVzU5yH-28cCf-ojGBemEziprBB=5uzby0OJsmUTkwNXJkiF=LEalaVMS=XMnJpPWIAhfBT7OGGB4OUMk=onoMa2GJnL4Wv4SsshAiEamEV7QKp53TIkQEXXpRXa3wvq0sjiXEH-0CGivZDiS2KK-ZgcB0d-_GMD4jifXUkEaw9xmMyNkAQ51HxxgRaqO1Io_2xVF=cz3xUIFN5dS044xhn2V2neeqHd31QL_xBAMvQd9En5rPmyHpy4tJGgYXqzOfznN87b_4xab=My0BkP8JC-=CduTE5yTzbZXpb3uAlhiVv=UvE1tCBoFGJgcqV_AJt_b80Re-2Tw0Jc-ELyVoNvFrVm4-eRFDNGYlRo0V_IRYaY3ImGsG4lyCjYRaTEFF1pI5=8e539paXrGCSQJlZ5qxcAh_454XM-frw_D=s_TiZLRYeFLdcvy1jgEbkKGEcj7DbuFxNu9_YEuto2PhPTnttYRB0XW3u-wYIfq0sEdkLVes3aEUgLg7UhVjJxKESVXZvoW2jRXgbNM0p0QnDtvC9lBz-z1U9=7UkFPLXAqG_DabIY920oEnlHpHXgDL9ifX5anNuNSvr25-BxXYRgxRnG9hah4eXAkc2PcBbqif5HA0nNxXvAzZ5j7ib=h-jqI8I84aDjehZTNxjIH=Z5gsYkTWVYO5U=cvyJ4Qe8gCJYzfDirhfR-I2lg7A5mvGJb15_tAjXLOnwI1874pJvhOUqfuD8eJpITuynP9LsqiRpEU1nw7X1s_Ray9SHU3EcGkr935P33m5CBvrNzASXSr=N7mzu-WxVjevBK9MrUNHNRfoXJfunTwGwPCYZEluL9YpTLDSZtrd470yVyZK-PHFo_TV8y=ETtA-IDysdrWrE9R-5DEQpkKpGJKGieNDHptrNMsVODRm2y_RGEzI_j-qmC8mQir2GDqTZKRWqmUzVkgCsIGviKYK5_h9JSERGlvrzkWIESKdX_skM-s1n0Q=Vm4bCmFz1hSBy3k2Dx0f8myZdLWR2E0InNHTjw5rmWELfr_=ax4l5EYJirE7sO3Wm9-eimauhNYen5KNC0PeGhJ7BLCu0eamN_PXge=r_G_HlF3xXhPA8FfkxCjV3v4fWCbtGBVIQGES0jQR0sGPmxHuZsQQ9BZuGhh9ykSYce-OmeOCu7xojRXh7PJx1AVwmhrm1jE48CTYSUfegx7p-CyeWKRkGOZ_gafCibxwnJ9mslopc0WviVvbbwBKrNuZrZlWXQrnzi8ml=oq35l4xWs0Nb7VZOYezEiUgoayg7pFm8eLV02GiU7lrpQMqF_pl04dXo3QAEwDW_YgIrLCtBoVM92dckSfScDL12nZREI3Yd-V3dkN23vFr2cFXbBKoHK2uVDTqUyzMxDgqHkXxPu3rzp9g0FSA95TgKke0ikBsX7KrdNBc9cQNgj2AouCXwIzzAExo2pa2wqJk7zXna3ObWHbgk1dpiJ=dk_Gbr3LOJ-CeTWGANr=u1Cnyo-weSvvLGMlQTJjWOedNDCtduC193Bfeqn2brGiKSUV3rx09ywipxjJVjGW=eJ_k0Cex7bIq_RNZJHso1qzQ-R=IR59XtaSyM9BLwKNJ4tOiCpMltCJyWOd0yJnYcgZSUCiLBINMh-e7F_Si-e_1c3VH5UqyENO8YvR59-IYl4qUQs5NpIEZc5Hl0lImENjL_7WsBB7jue4Ty0JPxXrYo3ME_P0Xc0UIEJKN3CQCzKl9a5ZmmMTrHyJHNNW0FSrMsAgUXRtDH_MuExGdqcEjgr-8hhoS2kAV_JER1FupukcsGi0JLegpLEYF9GLDnUoDCDrPRP5tQb21gQUpEdvozNmXdVNRCukPDJeRSSYr4iCM5=-3UzYPSDwkb=_7Ea=WvgXvKtIeKbYcF5ZPk8OU-rq_ysU2Slfaf7fgHHR4Aif2cx_ZTekL=0W5LQL3AZv2QBh-VjjNri
X-J3PopQvX-b: -1o1lvb
X-J3PopQvX-c: AEC_kwhyAQAA125LdUCpBmDn_gaT36bu5R1v7tLmaWD2Y_AhYfLAeVsrDVYa
X-J3PopQvX-d: o_2
X-J3PopQvX-f: AyT1lAhyAQAA645HTYkhB4gLDM5zokBx1RSq1NMZ34yfPEFY650x2JWUiytPAWf_BDWcuH01wH8AAEB3AAAAAA==
X-J3PopQvX-z: p
Host: api.sahibinden.com
Connection: Keep-Alive
Accept-Encoding: gzip

With each request I want to be able to regenerate following headers:

X-J3PopQvX-a
X-J3PopQvX-b
X-J3PopQvX-c
X-J3PopQvX-d
X-J3PopQvX-f 
X-J3PopQvX-z

I know of XMLHttpRequest but Im not sure how to use it in this case. I am new to node.js so a bit clueless

Edit: I am able to send my headers using this : try {

let res = await httpClient.get(url, { headers: headers, params: query, });

But I want to change j3pop headers everytime I send them. Need to change them manually. Is there any way for that?

Edit1: This code here generates new headers:

function() {
      'use strict';
      console.log("inside");
      var afterReadyCbCalled = false;
      var originalHeaders = ["X-Origin-DC", "gytp","Cache-Control", "max-age=0","X-Forwarded-For", "103.255.4.53","X-Client-SrcPort", "45244","Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","Accept-Language", "en-US,en;q=0.5","X-Forwarded-Proto", "https","X-TLS-Version", "771","Upgrade-Insecure-Requests", "1","ISTL-REFERER", "https://www.sahibinden.com/",];
      var originalBody = "";
      function afterReadyCb() {
        if (afterReadyCbCalled) return;
        afterReadyCbCalled = true;
        var xhr = new XMLHttpRequest();
        xhr.onload = function() {
          var isValid = xhr.getResponseHeader("ISTL-INFINITE-LOOP");
          if (isValid != null && isValid != '') return;
          var a = xhr.getResponseHeader("ISTL-REDIRECT-TO");
          if (a != null && a != '') {
            location.replace(a);
          } else {
            if (window.history != null && typeof history.replaceState === 'function') {
              var responseURL = xhr.responseURL != null ? xhr.responseURL : xhr.getResponseHeader("ISTL-RESPONSE-URL");
              if (responseURL != null && responseURL != '') {
                history.replaceState(null, '', responseURL);
              }
            }
            document.open();
            document.write(xhr.responseText);
            document.close();
          }
        };
        xhr.open("get", location.href, true);
        for (var i = 0; i < originalHeaders.length; i += 2) {
          var headerName = originalHeaders[i];
          try {
            xhr.setRequestHeader(headerName, originalHeaders[i + 1]);
          } catch (e) {}
        }
        xhr.setRequestHeader("ISTL-INFINITE-LOOP", '1');
        xhr.send(originalBody);
        var evt = document.createEvent('Event');
        evt.initEvent('QLpZFJdHv', true, true);
        dispatchEvent(evt);
      }
      addEventListener('afterReady', afterReadyCb, false);
      setTimeout(afterReadyCb, 400);
    }());

Code is extracted from website by simply saving the html. Here is the link to website : https://www.sahibinden.com/

What are you using in your backend? Express? It's important to know how to add headers to your response. — Daniel Cunha, Jun 03 '20 at 13:34
Maybe this is a duplicate of https://stackoverflow.com/questions/24113226/how-to-set-headers-in-node-js — Paolo Carrara, Jun 03 '20 at 13:36
@DanielCunha I am sending these requests to an API to scrape data — user3102085, Jun 03 '20 at 13:36
You want to send those headers to a nodejs server, or receive them from your nodejs server? — Paolo Carrara, Jun 03 '20 at 13:37
@PaoloCarrara neither. I'm sending headers to a URL of an API to receive response from it. Purpose of this is to scrape data — user3102085, Jun 03 '20 at 13:38
I got it. You are trying to send those headers to a server, and you are trying to do it using nodejs. So, just follow the link a added in the previous comment. — Paolo Carrara, Jun 03 '20 at 13:39
Sorry, the link will not help in your case. You need to send a http request from you node server. Here https://stackoverflow.com/questions/6158933/how-is-an-http-post-request-made-in-node-js#:~:text=Example%20code%3A,%2C%20err)%20return%20%7D%20console. — Paolo Carrara, Jun 03 '20 at 13:42
@PaoloCarrara but how do I add these specific j3pop headers? I don't want to send same headers. I want them to be generated with new values each time. — user3102085, Jun 03 '20 at 13:46
From the edited code I can infer that i must run this js code from this website "https://www.sahibinden.com/"? — Paolo Carrara, Jun 03 '20 at 16:13
This code was gotten from the website when it was opened in incognito window. It keeps sending requests and updates headers. There are two more functions but SO isn't letting me post them here — user3102085, Jun 03 '20 at 16:29
Sorry, I don't think I can keep helping you, it seems like the scope of the topic went from "How to X in code?" to "How to understand the security mechanisms of this site so I can scrap data from it?". Correct me if I'm wrong. — Paolo Carrara, Jun 03 '20 at 16:40
My question is that just like JS I can get a response and extract the header and use that header for next request like in the code, how can I do similar in Node.js? because apparently I can't call this JS file in node because it requires browser to render. — user3102085, Jun 03 '20 at 16:44

Paolo Carrara · Answer 1 · 2020-06-03T13:56:23.057

0

Here a sample code:

function doScrapRequest (postData, onSuccessCallback, onErrorCallback) {
    var options = {
        hostname: "your-host-name",
        port: 443,
        path: "/path-you-want-to-post-to",
        method: 'POST',
        headers: {
          'X-J3PopQvX-a': '0JMyqk3j...',
          'X-J3PopQvX-b': '-1o1lvb'
          'X-J3PopQvX-c': 'AEC_kwhy...'
          'X-J3PopQvX-d': 'o_2'
          'X-J3PopQvX-f': 'AyT1lA...'
          'X-J3PopQvX-z': 'p'
        }
    };

    var req = https.request(options, function (res) {
        res.setEncoding('utf8');

        var body = '';

        res.on('data', function (chunk) {
          body = body + chunk;
        });

        res.on('end',function(){
          console.log("Body :" + body);
          if (res.statusCode != 200) {
            onErrorCallback("Api call failed with response code " + res.statusCode);
          } else {
            // If you are expecting application/json you can do JSON.parse in the body variable.
            onSuccessCallback(body);
          }
        });
      });

      req.on('error', function (e) {
        console.log("Error : " + e.message);
        onErrorCallback(e);
      });

      // write data to request body
      req.write(postData);
      req.end();
}

edited Jun 03 '20 at 13:56

answered Jun 03 '20 at 13:48

Paolo Carrara

394
4
13

Will this generate new J3pop headers ? – user3102085 Jun 03 '20 at 14:05
I thought that when you said "I want to be able to regenerate following headers" you wanted to send back to the server these headers. But I'm now getting the impression that you want to generate new ones, is that right? – Paolo Carrara Jun 03 '20 at 14:08
Those headers are generated by the server that you are scrapping data, at some point the server sent this back to your application, you will have to identify when this happened and to this exact same thing every time you want new headers. There is no way for me or anyone to know how to regenerate those values without knowing more about the underlying server implementation of the headers. – Paolo Carrara Jun 03 '20 at 14:56
I have an html file that generates these headers. I extracted JS into a separate file. Would posting that here help? I couldn't understand it properly – user3102085 Jun 03 '20 at 15:05
It basically sends it to the server and gets those headers but I don't know how to convert that code – user3102085 Jun 03 '20 at 15:06
Post it, bad it can't do. – Paolo Carrara Jun 03 '20 at 15:26

How to add headers in node.js

1 Answers1