Node.JS spawning Python to process binary image gives error 'node error spawn ENAMETOOLONG'

Question

I'm trying to save an image by spawning a python process from a node.js script. The image is passed as a binary data file from a Node.JS script to a Python process.

The Script index.js spawns a python script my_py.py, then passes it a binary image data file. The python script my_py.py captures the binary image data and saves it to the directory assets/media.

The problem is the image isn't saved to the directory and I get the error error spawn ENAMETOOLONG.

Could you please help me spot the problem and to fix the code?

Thanks in advance!

index.js:


const spawn = require("child_process").spawn;
const fs = require('fs');

let params = {
    "image":readDataset()
  }

// Launch python app
const pythonProcess = spawn('py',['my_py.py', JSON.stringify(params)]);

// Print result
pythonProcess.stdout.on("data", (data) =>{
  console.log(data.toString());
});

// Read errors 
pythonProcess.stderr.on("data", (data) =>{
   console.log(data.toString());
});

function readDataset() {
    try { 
        return fs.readFileSync('color.png',  'binary');
    }
    catch (err) { 
        return err;
    }
}

my_py.py:


import sys, json

params = json.loads(sys.argv[1]) # Load passed arguments

import numpy as np
import cv2
import os

fileName = "image.png" # file name
fileData = params['image'] # Capture file

# Convert Image to Numpy as array
img = np.array(fileData)

# Save file to local directory
cv2.imwrite(os.path.join('assets/media/', f'{fileName}'), img)
cv2.waitKey(0)          

# Results must return valid JSON Object
print("File saved!")
sys.stdout.flush()

You are passing an image (which can be multiple megabytes) as a command line argument. Most command lines have a maximum input size of thousands of bytes. For example Linux typically can process a command 4096 bytes long and you are passing the encoded image data as command line argument. That is way longer than 4000 bytes!! Use stdin/stdout to pass your image data — slebetman, Mar 29 '22 at 07:14
Windows is a bit better allowing you to construct commands 8191 bytes long. Mac OSX can accept commands 250k long. So I guess for small images if you really want to do this you need to buy a Mac — slebetman, Mar 29 '22 at 07:16
Also it's not a good idea to pass binary data as command line argument. You need to base64 encode the image for safe delivery over a text channel — slebetman, Mar 29 '22 at 07:20

slebetman · Accepted Answer · 2022-03-29T11:36:28.870

The error says it all: the command you are constructing is too long.

The thing you need to be aware of is that operating systems limits how long a command can be. For Linux this is around 4096 bytes though you can modify this value. For Windows this is 8191 bytes and for Mac OS this is around 250k bytes.

Part of the reason for this is because these OSes were written in C/C++ and in C/C++ code, not enforcing buffer size limit is an invitation to buffer overrun (the infamous stack overflow or underflow bug!!). Truly making input size unlimited result in slow code because you will not be using simple arrays for input buffer in that case but more complicated data structures.

Additionally, not having any limit on command length is a vector for DOS attacks. If you run an OS that does not have command size limit and I know for sure you have 32GB of RAM all I need to do to crash your system is construct a 32GB command!!

TLDR

The correct way to pass large data between processes is what you've been doing over the internet - upload/download the data! The simplest implementation is to just pass the data via the stdin of the python process you are connected to:

const spawn = require("child_process").spawn;
const fs = require('fs');

let params = {
    "image":readDataset()
  }

// Launch python app
const pythonProcess = spawn('py',['my_py.py']);

// Pass image data
pythonProcess.stdin.write(JSON.stringify(params) + '\n');

// Print result
pythonProcess.stdout.on("data", (data) =>{
  console.log(data.toString());
});

// Read errors 
pythonProcess.stderr.on("data", (data) =>{
   console.log(data.toString());
});

function readDataset() {
    try { 
        return fs.readFileSync('color.png',  'base64');
    }
    catch (err) { 
        return err;
    }
}

In the code above I end the JSON "packet" with a newline so that the python code can read until end of line for a single packet. You can use any convention you like. For example I also often use the nul character (0x00) to mark end of packet and HTTP use two newlines ('\n\n') to mark end of header etc.

In any case I read the image file as base64 because binary data is invalid in JSON. In your python code you can do a base64 decode to get the image back. Additionally base64 does not include newlines ('\n') in its character set so converting the image to base64 ensure you don't get a newline inside your JSON data.

To get the image just read from stdin:

import sys, json, base64

params = json.loads(sys.stdin.readline())

import numpy as np
import cv2
import os

fileName = "image.png" # file name
fileData = base64.b64decode(params['image']) # Capture file

# Convert Image to Numpy as array
img = np.array(fileData)

# Save file to local directory
cv2.imwrite(os.path.join('assets/media/', f'{fileName}'), img)
cv2.waitKey(0)          

# Results must return valid JSON Object
print("File saved!")
sys.stdout.flush()

Just read from stdin the normal way. I don't know python but a bit of googling gave me this: https://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin. In C/C++ reading from stdin is among the first things you learn: `gets()` — slebetman, Mar 29 '22 at 11:28

Node.JS spawning Python to process binary image gives error 'node error spawn ENAMETOOLONG'

1 Answers1

TLDR