Unexpected end of input when sending request from nodejs to php

Question

So, I have looked at other answers with the same error but none of their solutions have answered my question.

I have a mysql table with ~10k names in it and I have 25 instances (listening on different ports) of a nodejs file that each send a request to grab 400 names from database through a php file. If I set the request to grab like 100 - 150 it works but if I tell it to get more than 200, I get the error 'unexpected end of input'.

This is my nodejs request:

function user_acquire()
{ 
var request = require('request');
var http = require('http');  

    var post_options = {
        host: 'localhost',
        path: '/name_request.php?pos1=50&pos2=150', //this is where the amount to request is set & range
        method: 'POST',
        headers: {
            'User-Agent': 'Super Agent/0.0.1',
            'Content-Type': 'application/x-www-form-urlencoded',
        }
    };            


// Set up the request
    var post_req = http.request(post_options, function (res) {
        res.on('data', function (chunk) {
            usernames = JSON.parse(chunk);
        });
    });
    post_req.end(); 
}

And this is the php file the nodejs communicates with:

<?php

$servername = "localhost"; 
$username = "root"; 
$password = ""; 
$dbname = "test"; 

$conn = new mysqli($servername, $username, $password, $dbname, '3306');
if ($conn->connect_error) 
{
    die("Connection failed: " . $conn->connect_error);
}

$start = $_GET['pos1'];
$end = $_GET['pos2'];

$sql = mysqli_query($conn, "SELECT * FROM preliminary_tests WHERE ID BETWEEN   '$start' AND '$end'");
$array = mysqli_fetch_all($sql, MYSQLI_ASSOC);
print_r(json_encode($array,JSON_UNESCAPED_UNICODE));

?>

using JSON_UNESCAPED_UNICODEdoesn't fix the problem. I know the mysql is encoded in utf-8. Both of these functions work with smaller requests but increasing the request size breaks one and I'm not sure why

Perhaps after 200 usernames the response becomes too large and is split up into multiple chunks when sent over to node. It looks like you're parsing each chunk on arrival, but it may that the chunk cuts off the JSON prematurely, which is why you get the error. You should be able to append each chunk on arrival in the response's 'data' event, and parse them all when the response ends in its 'end' event. — Stiliyan, Feb 05 '16 at 13:29
@Stiliyan How would you suggest I append them through the data event, I have a rough idea? — Miniman, Feb 05 '16 at 21:44
You should be able to use a closure by declaring `var usernames = "";` just above the `res.on('data')` callback. `usernames` would now be in scope inside the 'data' callback, where you can append the incoming chunk `usernames += chunk;`. Then, listen for the response's end event, and inside its callback parse the usernames. Something like: `res.on("end", function() { usernames = JSON.parse(usernames);})` — Stiliyan, Feb 05 '16 at 22:44
@Stiliyan Is this what you meant? `var post_req = http.request(post_options, function (res) { var usernames= " "; res.on("end", function () { usernames = JSON.parse(usernames); }); });` — Miniman, Feb 06 '16 at 01:05
You would need both the data and the end callbacks. I decided to turn my comments into an answer. Hope it's useful and good luck! — Stiliyan, Feb 06 '16 at 02:42

score 3 · Accepted Answer · edited May 23 '17 at 12:10

After 200+ usernames the response becomes too large and is split up into multiple chunks when it is sent over to node. It looks like you're parsing each chunk on arrival, but it may be that the chunk cuts off the JSON prematurely, which is why you get the error. You should be able to append each chunk on arrival in the response's 'data' event, and parse them all in the response's 'end' event.

This would look something like:

// Set up the request
var post_req = http.request(post_options, function (res) {
    // create an enclosure for usernames 
    // so it's in scope in both callbacks,
    // and instantiate it as an empty string:
    var usernames = "";  

     // append each chunk on arrival:
    res.on('data', function (chunk) {
        usernames += chunk;
    });

    // at the end of the response, parse the full set of chunks:
    res.on('end', function() {
        usernames = JSON.parse(usernames);
        // do something with the parsed usernames
    });
});
post_req.end();

Take a look at closures on MDN or CodeSchool, because they are a useful tool when you want to share scope between callbacks.

Also, check Node's Docs on HTTP Response's "chunk" and "end" events to learn more ways to use them.

Update: Here is a great explanation on closures on SO.

Thank you kindly!!!! Sorry I had to ask a few times, I've only learnt javascript through this project I'm doing currently — Miniman, Feb 06 '16 at 11:13
You're welcome! No problem at all - it was a well asked question. Enjoy learning JavaScript! — Stiliyan, Feb 06 '16 at 11:28

score 1 · Answer 2 · answered Feb 05 '16 at 13:24

1

You may have this behaviour due to they way u passing the variables $start and $end. In the current situation, the query is looking for strings in the database, but mostly probably ID is a an integer. This might is explain this behavior.

If it is as I assume, then cast the two variables as :

$start = (int)$_GET['pos1'];
$end = (int)$_GET['pos2'];

and then change the sql to :

"SELECT * FROM preliminary_tests WHERE ID BETWEEN   $start AND $end"

answered Feb 05 '16 at 13:24

ArrowHead

609
5
16

ID is indeed and int, trying this didn't remedy the problem for large ones but ty – Miniman Feb 05 '16 at 21:41

Unexpected end of input when sending request from nodejs to php

2 Answers2