Reading a binary file from memory in chunks of 10 bytes with python

Question

I have a very big .BIN file and I am loading it into the available RAM memory (128 GB) by using:

ice.Load_data_to_memory("global.bin", True)

(see: https://github.com/iceland2k14/secp256k1)

Now I need to read the content of the file in chunks of 10 bytes, and for that I am using:

with open('global.bin', 'rb') as bf:
    while True:
        data = bf.read(10)
            if data = y:
                do this!

This works good with the rest of the code, if the .BIN file is small, but not if the file is big. My suspection is, by writing the code this way I will open the .BIN file twice OR I won't get any result, because with open('global.bin', 'rb') as bf is not "synchronized" with ice.Load_data_to_memory("global.bin", True). Thus, I would like to find a way to directly read the chunks of 10 bytes from memory, without having to open the file with "with open('global.bin', 'rb') as bf"

Looking at the linked library it seems that the `Load_data_to_memory` is very specific to its use (whatever it is that `ice.check_collision` does) and not a general purpose way to load data in memory. You shouldn't expect to be able to access the data loaded in this way — Aaron, Feb 06 '23 at 11:04
I'm using this methode to get collisions, if a match is found. — user2958297, Feb 06 '23 at 11:06

user2958297 · Answer 1 · 2023-02-26T12:05:59.037

I found a working approach here: LOAD FILE INTO MEMORY This is working good with a small .BIN file containing 3 strings of 10 bytes each:

 with open('0x4.bin', 'rb') as f:
  # Size 0 will read the ENTIRE file into memory!
  m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) #File is open read-only

  # Proceed with your code here -- note the file is already in memory
  # so "readine" here will be as fast as could be
  data = m.read(10)  # using read(10) instead of readline()
  while data:
     do something!

Now the point: When using a much bigger .BIN file, it will take much more time to load the whole file into the memory and the while data: part starts immediately to work, so I would need here a function delay, so that the script only starts to work AFTER the file is completely loaded into the memory...

So, as a finally conclusion and solution for my own question, I came across the possibility given by the refered python ICE library:

list = [bytes.fromhex(line.split()[0]) for line in open("myfile.txt",'r')]
_bits, _hashes, _bf = ice.Fill_in_bloom(list, 0.00000000000000000001)  
del list
ice.dump_bloom_file("myfile.bin", _bits, _hashes, _bf)
_bits, _hashes, _bf = ice.read_bloom_file("myfile.bin")

To check for collisions, just use the condition:

this_check = xyz
if ice.check_in_bloom(this_check, _bits, _hashes, _bf) == True:
    DO SOMETHING!

Reading a binary file from memory in chunks of 10 bytes with python

1 Answers1