1

I have a byte string returned from API and store in response.content

With small content, I can save it into a file with no problem using the following code

with open(save_path, 'wb') as save_file:
    save_file.write(response.content)

But for the larger file, it will cause Memory Error, So I tried not to read the content all at once, by using this code

with open(save_path, 'wb') as save_file:
    for x in response.content: 
        save_file.write(bytes(x)) #the x from iteration seem to be converted to int so I convert it back

But the method above seems to alternate the content because it doesn't compatible with another library anymore (In my when Laspy try to read the saved file, laspy.util.LaspyException: Invalid format: h0.0 error appears)

How can I do it?

BombShelley
  • 107
  • 9
  • 1
    Try changing `'wb'` to `'ab'` and see what happens. It looks like every time you write the next part, you are erasing the rest of it, and should be in `'a'` (append) mode instead of `'w'` (write) mode. – Fricative Melon Feb 19 '21 at 03:22
  • 1
    You might open file results to see what happened in 2 ways for easier checking – Tấn Nguyên Feb 19 '21 at 03:32
  • Thanks, Fricative Melon and Tấn Nguyên for suggestions. I tried changing it to 'ab' as you said. But the problem still occurs. – BombShelley Feb 19 '21 at 04:06

1 Answers1

1

I see your problem on using bytes(x). change it to x.to_bytes(1, 'big') solve your problem

Use below code to reveal what difference

a = b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'
a.decode('utf-8') # τoρνoς

with open('./save.txt', 'wb') as save_file:
    for i in a:
        print(i.to_bytes(1, 'big')) # write it to file, not the others
        print(i)
        print(bytes(i))
        print('----')

enter image description here

Tấn Nguyên
  • 1,607
  • 4
  • 15
  • 25