Python: OverflowError long int too large to convert to int

Question

After many years finding answers here, it's finally my time to ask my first question ever.

Running a RPI3B+ like a mini-server at home, I want to ba able to send large files to friends and family over internet, via CLI. To do this, I am using this : https://github.com/justbeamit/beam/blob/master/beam

But, when I want to upload a file larger than ~1.5Gb (by my estimate), I get an error saying :

OverflowError
long int too large to convert to int

After a short investigation, I can see that it comes from the line 288 where the max value of the progress bar is set, in this method :

def transfer(token, filePaths):

  print("recipient has connected! starting transfer...")

  uploadUrl = ACTIVE_BACKEND + "/upload"

  try:

    index = 0
    ProgressBar.totalNumberOfFiles = len(filePaths)

    for filePath in filePaths:

      # make the console look pretty
      sys.stdout.write("\n")
      print("  " + filePath)

      # the callback function invoked by the monitor
      def updateProgress(monitor):
        theProgressBar.update(monitor.bytes_read)


      # setup the multi-part encoder & its monitor
      fileMPE = getMultipartEncoder(filePath)
      monitor = MultipartEncoderMonitor(fileMPE, updateProgress)

      # setup the progress bar
      ProgressBar.fileNumber = index + 1 # to avoid showing (0 of 3)

      # since the progress bar will be updated by the multi-part encoder, we can't set 'maxval'
      # to be the file's size since the encoder adds extra bytes to account for the header
      theProgressBar = ProgressBar(
        maxval = len(fileMPE), 
        widgets = WIDGETS,
      )

      theProgressBar.start()

      urlParams = {
        "type": "CLI",
        "token": token,
        "index": index
      }

      requests.post(
        uploadUrl,
        data=monitor,
        params=urlParams,
        headers={'Content-Type': monitor.content_type}
      )

      theProgressBar.finish()
      index += 1
    # end for loop

  except Exception as e:
    print(e.__class__.__name__)
    print(e)
    exit()

  finally:
    sys.stdout.write("\n")

Can anyone help me? This is annoying because without the progress bar, everything would work perfectly fine. I tried commenting this line, but then the error moves somewhere else, at line 300 (requests.post).

My infos :

python --version => Python 2.7.13

Raspbian version => PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"

Ionut Ticus · Accepted Answer · 2020-05-09T08:49:04.610

0

Are you sure the combined file size is not larger than 4 GB? I think you're hitting this Python "bug" as your Pi probably runs 32bit.
The issue is mentioned in requests_toolbelt's code as well.
To fix this you need a more recent version of requests_toolbelt (I tested successfully with 0.6.0) and a small change to beam itself:

theProgressBar = ProgressBar(
    maxval = fileMPE.len,
    widgets = WIDGETS,
)

edited May 09 '20 at 08:49

answered May 06 '20 at 16:04

Ionut Ticus

2,683
2
17
25

Thanks for that! But now the error is the same but later in the program. It happens in the "requests.post" part. And yes, I tried with files between 1Gb and 4Gb. – Victortuga May 06 '20 at 18:48
Can you post the full traceback for `requests.post` exception? – Ionut Ticus May 07 '20 at 13:01
Where do I find it? It just says "OverflowError long int too large to convert to int" I know it is in the requests.post because I used "print("test") a cross the program to find out where it crashes. Otherwise I'm in the blind.. :/ – Victortuga May 08 '20 at 14:08
The traceback is the complete error message you get (it should include line numbers and files on which the error occurred). – Ionut Ticus May 08 '20 at 15:53
Thanks, but where do I find it? On my console it only shows the Overflow error, and that's it. – Victortuga May 08 '20 at 16:30
Here is a pastebin of a traceback attempt : [https://pastebin.com/4WTcUink](https://pastebin.com/4WTcUink) – Victortuga May 08 '20 at 16:49
Here is another one with only n(ext) commands during debug. [https://pastebin.com/r9LZaVYz](https://pastebin.com/r9LZaVYz) -- FYI, I added `print("test 7")` between "urlParams" (l.294) and "requests.post" (l.300) – Victortuga May 08 '20 at 17:01
The full message wasn't shown because the Exception was caught and only the class and message was printed; I debugged the code a little and found a solution (see updated answer). – Ionut Ticus May 09 '20 at 08:58

Python: OverflowError long int too large to convert to int

1 Answers1