2

Using express and Node.js, I'm using the twitter streaming API and the needle npm package for accessing APIs to pull tweets related to keywords. The streaming is functional and I am successfully pulling tweets using the following (simplified) code:

const needle = require('needle');
const TOKEN = // My Token 
const streamURL = 'https://api.twitter.com/2/tweets/search/stream';

function streamTweets() {

    const stream = needle.get(streamURL, {
        headers: {
            Authorization: `Bearer ${TOKEN}`
        }
    });

    stream.on('data', (data) => {
        try {
            const json = JSON.parse(data); // This line appears to be causing my error
            const text = json.data.text;
        } catch (error) {
             console.log("error");
        }
    });
}

However, no matter which search term I use (and the subsequent large or small volume of tweets coming through), the catch block will consistently log 1-3 errors per minute, which look like this:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at PassThrough.<anonymous> (C:\Users\danie\OneDrive\Documents\Personal-Projects\twitter-program\server.js:56:31)
    at PassThrough.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at PassThrough.Readable.push (internal/streams/readable.js:223:10)
    at PassThrough.Transform.push (internal/streams/transform.js:166:32)
    at PassThrough.afterTransform (internal/streams/transform.js:101:10)
    at PassThrough._transform (internal/streams/passthrough.js:46:3)
    at PassThrough.Transform._read (internal/streams/transform.js:205:10).

I've seen previous advice which says that data can be fired in multiple chunks, and to push the chunks to an array i.e. something like the following:

let chunks = [];
        stream.on('data', (dataChunk) => {
chunks.push(dataChunk);
        }).on('end',() => {
// combine chunks to create JSON object
})

But this didn't work either (may have been my implementation but I don't think so) and now I'm wondering if it's perhaps an error with the twitter API, because most of the tweet objects do come through correctly. I should note that the streamTweets() function above is called from an async function, and I am also wondering if that is having something to do with it.

Has anyone else encountered this error? Or does anyone have any idea how I might be fix it? Ideally i'd like 100% of the tweets to stream correctly.

Thanks in advance!

Daniel Hill
  • 21
  • 1

1 Answers1

0

For future readers, this error is triggered by Twitter's heartbeat message that is sent every 20 seconds. Per the documentation:

The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character).

Adding a guard against parsing the empty string will prevent the JSON parsing error.

if (data === "")
  return

An empty string is invalid JSON, hence the emitted error.

Now, acknowledging that the heartbeat exists, it may be beneficial to add read_timeout = 20 * 1000 in the needle request to avoiding a stalled program with no data, be that due to a local network outage or DNS miss, etc.

alanbixby
  • 11
  • 3