-1

My requirement is to split a big file (e.g. 500MB)size into small files (50MB) in python.

What are the modules i can use in python to achieve this?

For eg. I have a file of 500MB size i want to split that file into 10 50 MB files and send it to an API

Thanks in advance.

Er. Bikram Keshari
  • 1
  • 1
  • 4

1 Answers1

0

You don't need any external modules. (In fact, you don't need to even import anything.)

This would chop up large_file.dat into 50-megabyte pieces and write them to disk – but you could just as well replace the file writing with whatever API call you need.

filename = "large_file.dat"
chunk_size = 50_000_000  # bytes; must fit in memory
chunk_num = 1

with open(filename, "rb") as input_file:
    while True:
        chunk = input_file.read(chunk_size)
        if not chunk:  # Nothing more to read, we've reached file end
            break
        with open(f"{filename}.{chunk_num:04d}", "wb") as output_file:
            output_file.write(chunk)
        chunk_num += 1
AKX
  • 152,115
  • 15
  • 115
  • 172
  • Thank you @AKX. That looks like a good solution to my problem. But it seems you have missed the for loop in the code you have shared – Er. Bikram Keshari Jul 30 '21 at 10:23
  • @Er.BikramKeshari Good catch. It was there in an earlier revision, and I simplified things and dropped it. Fixed now. :) – AKX Jul 30 '21 at 10:25
  • Hi @AKX, My requirement - reading a big file stored in a AWS S3 bucket and split that file into small chunks. And also i need to execute this snippet of code from a lambda function. Do you have any idea how we can do that? – Er. Bikram Keshari Jul 30 '21 at 19:34
  • https://stackoverflow.com/a/40661459/51685 shows you how to read an S3 file in chunks using boto3. Boto3 can also be used to write the smaller files back to S3. – AKX Jul 30 '21 at 19:45
  • AKX, really appreciate your quick response. I will go through the link. Thank you. – Er. Bikram Keshari Jul 30 '21 at 20:02