How do I get data out of a Node http(s) request?

Question

How do I get the data from a https request outside of its scope?

Update

I've seen Where is body in a nodejs http.get response?, but it doesn't answer this question. In fact, that question isn't answered accurately, either. In the accepted answer (posted by the asker), a third party library is used. Since the library returns an object different from that returned by http.get() it doesn't answer the question.

I tried to set a variable to the return value of http.get() using await, but that returns a http.clientRequest and doesn't give me access to the response data that I need.

I'm using Node v8.9.4 with Express and the https module to request data from Google's Custom Search.

I have two routes. One for a GET request and one for a POST request used when submitting a form on the front page. They both basically serve the same purpose... request the data from CSE and present the data as a simple JSON string. Rather than repeat myself, I want to put my code for the CSE request into a function and just call the function within the callback for either route.

I thought about returning all the way up from the innermost callback, but that won't work because it wouldn't get to the request's error event handler or the necessary .end() call.

Here's a subset of the actual code:

app.get('/api/imagesearch/:query', newQuery)
app.post('/', newQuery)

function newQuery (req, res) {
  let query = req.body.query || req.params.query
  console.log(`Search Query: ${query}`)

  res.status(200)
  res.set('Content-Type', 'application/json')

  // This doesn't work
  let searchResults = JSON.stringify(cseSearch(req))
  res.end(searchResults)
}

function cseSearch (request) {
  let cseParams = '' +
    `?q=${request.params.query}` +
    `&cx=${process.env.CSE_ID}` +
    `&key=${process.env.API_KEY}` +
    '&num=10' +
    '&safe=high' +
    '&searchType=image' +
    `&start=${request.query.offset || 1}`

  let options = {
    hostname: 'www.googleapis.com',
    path: '/customsearch/v1' + encodeURI(cseParams)
  }

  let cseRequest = https.request(options, cseResponse => {
    let jsonString = ''
    let searchResults = []

    cseResponse.on('data', data => {
      jsonString += data
    })

    cseResponse.on('end', () => {
      let cseResult = JSON.parse(jsonString)
      let items = cseResult.items
      items.map(item => {
        let resultItem = {
          url: item.link,
          snippet: item.title,
          thumbnail: item.image.thumbnailLink,
          context: item.image.contextLink
        }
        searchResults.push(resultItem)
      })

      // This doesn't work... wrong scope, two callbacks deep
      return searchResults
    })
  })

  cseRequest.on('error', e => {
    console.log(e)
  })

  cseRequest.end()
}

If you're curious, it's for a freeCodeCamp project: Image Search Abstraction Layer

Possible duplicate of [Where is body in a nodejs http.get response?](https://stackoverflow.com/questions/6968448/where-is-body-in-a-nodejs-http-get-response) — Behrooz, Apr 06 '18 at 04:59
I don't really understand what the problem is. You can put common code into a function and call the common code from two different routes. A route is no different than any other Javascript function. Put shared code into a function and call that function from more than one place. The key here is that you put ONLY the common code into the shared function. Your POST handler and your GET handler don't do exactly the same thing so you can't use exactly the same code for both so you can't use the exact same function for both. But, you can factor common code into a shared function. — jfriend00, Apr 06 '18 at 05:03
@jfriend00 I'm sure I'm missing something... If you see an easy way to do this, please post an answer with an example. In my example code I did put common code into a function and call it, but I don't know how to get the information. I can't use Promises and `.then()` because of the way the `data` event has to be handled. There's no indication that `https.request()` returns a Promise anyway. I can't just return the information in each callback and function because `request.end()` has to be called after defining handlers for the `end` and `error` events. — Vince, Apr 06 '18 at 05:26
@Vince - Use the `request-promise` library instead of `https.request` and return a promise. Then, use `.then()` on the returned promise from the caller. This is how you communicate back an asynchronous result from a function. You won't call `JSON.stringify(cseSearch(req))` then. Instead, you'll do `cseSearch(req).then(function(result) res.end(JSON.stringify(result))).catch(...)`. — jfriend00, Apr 06 '18 at 05:29
@jfriend00 That probably would have worked, but I had the feeling that installing another module for this was overkill and my primary goal isn't just to get the job done, it's to learn more about how all this stuff works. Salman's solution worked perfectly (with a little tweaking), so I guess my feeling was right :) — Vince, Apr 06 '18 at 09:16
Learning is fine. But, one of the BIG reasons for using node.js is the huge library of already solved modules in the NPM library. That is part of what you need to learn too. It's never overkill to use an already documented and tested module that does exactly what you need. For example, I never ever use `http.request()` or `http.get()` because the `request` and `request-promise` libraries are soooo much easier to use. — jfriend00, Apr 06 '18 at 15:28
As a further reason to use existing modules, Salman's code example has no error handling for the promise. If the request fails, what happens? The request-promise library has fully tested error handling. — jfriend00, Apr 06 '18 at 15:28
@jfriend00 I'm paying attention I've been reading the docs for [`request-promise`](https://www.npmjs.com/package/request-promise), [`request-promise-native`](https://www.npmjs.com/package/request-promise-native), and [`request`](https://www.npmjs.com/package/request). I'm happy with this project in its current state and I want to keep moving forward, but I'm probably going to be using `request-promise-native` in future projects. Thank you. — Vince, Apr 06 '18 at 20:15

score 3 · Accepted Answer · answered Apr 06 '18 at 06:32

using promise method solve this issue.

cseSearch(req).then(searchResults=>{
    res.end(searchResults)
   }).catch(err=>{
     res.status(500).end(searchResults)
   })
   function cseSearch (request) {
     return new Promise((resolve, reject)=>{
     ...your http request code 
       cseResponse.on('end', () => {
           let cseResult = JSON.parse(jsonString)
           let items = cseResult.items
           items.map(item => {
               let resultItem = {
               url: item.link,
               snippet: item.title,
               thumbnail: item.image.thumbnailLink,
               context: item.image.contextLink
              }
              searchResults.push(resultItem)
            })
            resolve(searchResults);
       })
    })
  }

This shows my inexperience with Node and ES6. I knew I needed a Promise, but `https.request()` doesn't return one. It didn't even occur to me to create my own promise. What I did was very similar to your solution. If you're curious, you can see it in context at https://github.com/VAggrippino/imageSearch — Vince, Apr 06 '18 at 09:11

jfriend00 · Answer 2 · 2018-04-07T04:52:03.320

1

Based on what I explained in the comments, to give you an idea how compact your code could be using the request-promise library, here's what you could use:

const rp = require('request-promise-native');

app.get('/api/imagesearch/:query', newQuery)
app.post('/', newQuery)

function newQuery (req, res) {
  let query = req.body.query || req.params.query
  console.log(`Search Query: ${query}`)

  cseSearch(req).then(results => {
      res.json(results);
  }).catch(err => {
      console.log("newQueryError ", err);
      res.sendStatus(500);
  });
}

function cseSearch (request) {
  let cseParams = '' +
    `?q=${request.params.query}` +
    `&cx=${process.env.CSE_ID}` +
    `&key=${process.env.API_KEY}` +
    '&num=10' +
    '&safe=high' +
    '&searchType=image' +
    `&start=${request.query.offset || 1}`

  let options = {
    hostname: 'www.googleapis.com',
    path: '/customsearch/v1' + encodeURI(cseParams),
    json: true
  };

  return rp(options).then(data => {
    return data.items.map(item => {
        return {
            url: item.link,
            snippet: item.title,
            thumbnail: item.image.thumbnailLink,
            context: item.image.contextLink
        };
    });
  });

edited Apr 07 '18 at 04:52

answered Apr 07 '18 at 04:04

jfriend00

683,504
96
985
979

Thank you. This is clearly a much better way. Salman's solution is still my answer as he answered the question within the context that I asked it. You showed me a better way (I see you catching and properly handling the error I neglected, too), but it's not the built-in https module I asked about. I'm going to use `request-promise-native` going forward, but I'm not going back in to fix the code for this project. I want to keep moving forward at a steady pace. If I go into my practice projects to make improvements every time I learn a better way I'll never make any progress. – Vince Apr 07 '18 at 04:53
@Vince - OK, no problem. Just wanted to show you this way. You can decide how to spend your "best answer" vote however you want. Some people choose to vote the answer that was of most value to you or will be of most value to other readers. I try to understand the overall problem being solved and come up with the best way to solve the problem regardless of how the current attempt is written. That happens a lot here because people ask questions about problems with their particular solution which isn't always the path to the best possible solution. Anyway, glad you got it working. – jfriend00 Apr 07 '18 at 05:09

How do I get data out of a Node http(s) request?

Update

2 Answers2