OSError: [Errno 7] Argument list too long on ubuntu, python calling bitcoind-cli with popen

Question

Running a python script calling bitcoind-cli using popen on ubuntu, on large blocks with many trasactions, when calling getrawtransaction i get the error OSError: [Errno 7] Argument list too long

i understand it's a buffer issue between the shell and the python script? there's a single argument, i guess it's just a very long one

need i check something else? can i make the buffer larger somehow or should i change the method i interact with bitcoind to RPC?

tried it on a local and an AWS ubuntu machines

thanks

score 1 · Answer 1 · edited Apr 13 '17 at 12:36

It is your OS limitation e.g.:

>>> import os
>>> os.execl('/bin/ls', 'ls', 'c'*10**7)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/os.py", line 314, in execl
    execv(file, args)
OSError: [Errno 7] Argument list too long

Is it necessary to pass the data on the command line (could you use a pipe/file/socket, etc instead?) in your case? Can you run the command several times with splitted command-line arguments? See Solving “mv: Argument list too long”.

You may get the same error if the passed environment is too large:

>>> os.execle('/usr/bin/env', 'env', {'envvar': 'c'*10**9})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/os.py", line 322, in execle
    execve(file, args[:-1], env)
OSError: [Errno 7] Argument list too long

The solution is to sanitize the passed environment to avoid unused large envvars.

The limits might be hardcoded in the kernel source.

score 0 · Answer 2 · answered Mar 07 '16 at 21:04

Since you are using Python, the best thing you can do is use RPC, such as:

import base64
import requests

response = requests.post(
    bitcoind_url, 
    data=json.dumps(
        {
            'method': method,
            'params': params,
            'jsonrpc': '2.0',
            'id': 0,
        }
    ), 
    headers={'content-type': 'application/json', 'Authorization': b'Basic ' + base64.b64encode(rpcuser + b':' + rpcpassword)})

where params is a list of the arguments for the specific method.

You can get rpcuser and rpcpassword from the bitcoind configuration file.

OSError: [Errno 7] Argument list too long on ubuntu, python calling bitcoind-cli with popen

2 Answers2