0

He or she who answers this question will have a very special place in my heart. I'm trying to update many records at once using Curl with PHP on elasticsearch 7.13. It seems that no matter what I do, I get the error "The bulk request must be terminated by a newline". I have tried many, many renditions of this code. Is there perhaps a way to overload what PHP CURL is doing to force it to use binary data? I believe it is stripping the newline characters, but if I change to plain text submission it won't be accepted by elasticsearch.

    public function bulkUpdate(){
        $query_json_store = '';
        foreach($this->records['hits']['hits'] as $person){
        
            $entry1 = [
                "update"=>[ 
                    "_id"=> $person['_id'],
                    "type"=> "doc",
                    "_index"=> "human_data",
                ]
                ];
            
                $entry2 = [
                    "doc"=>[
                        "scanned"=> true
                    ]
                ];              


        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, 'localhost:9200/human_data/doc/_bulk');
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
        curl_setopt($ch, CURLOPT_POSTFIELDS, '\n'.json_encode($params)).'\n';
        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
        $headers = array();
        $headers[] = 'Content-Type: application/x-ndjson';
        curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        $result = curl_exec($ch);
        if (curl_errno($ch)) {
            echo 'Error:' . curl_error($ch);
        }
        curl_close($ch);
        \Log::info($result);
        }
    }
Christopher Rox
  • 7
  • 2
  • The bulk endpoint does not expect a proper JSON body, but rather a multi-line body where each line contains a JSON object (e.g. line 1: the command, line 2: the document, line 3: next command, line 4: next document, etc.) where each line is terminated by a newline character (which must be in *double* quotes in PHP, i.e. `"\n"` instead of `'\n'`). See [this answer](https://stackoverflow.com/a/33340234/1941241) for an example. You will not get that request body from `json_encode()`. – rickdenhaan Jun 23 '21 at 19:52
  • @rickdenhaan great answer – Val Jun 23 '21 at 19:55
  • Christopher it's not clear how you construct your `$params` but as Rick stated, you cannot use `json_encode`. See the answer Rick pointed at for how the body should be formatted – Val Jun 23 '21 at 19:58
  • The problem is that PHP passes through new lines incorrectly with single quotes. I through doubles over it and it works. – Christopher Rox Jun 24 '21 at 00:48

0 Answers0