Stack Overflow
Stack Overflow

Questions tagged [memmap]

21 questions
2
votes
2 answers

big data dataframe from an on-disk mem-mapped binary struct format from python, pandas, dask, numpy

I have timeseries data in sequential (packed c-struct) format in very large files. Each structure contain K fields of different types in some order. The file is essentially an array of these structures (row-wise). I would like to be able to mmap…
Jonathan Shore
  • 880
  • 2
  • 8
  • 21
2
votes
1 answer

Nim: work with read-only memory mapped files

I've only just started with Nim, hence it possibly is a simple question. We need to do many lookups into data that are stored in a file. Some of these files are too large to load into memory, hence the mmapped approach. I'm able to mmap the file by…
Juergen
  • 699
  • 7
  • 20
2
votes
0 answers

Can I extract some data from hex-table (memmap) by python?

I have an emulator of Game Boy called BGB and Pokemon Red/Blue game. In the emulator, I can open a debug window. And in the window I can see the entire memory of the game in a hex-table (memmap): The part of the window which I need is located in the…
ForgottenGhost
  • 29
  • 1
2
votes
1 answer

Python: numpy memmap in parallel

Does it make sense to use numpy's memmap across multiple cores (MPI)? I have a file on disk. Can I create a separate memmap object on each core, and use it to read different slices from the file? What about writing to it?
Tom
  • 223
  • 1
  • 2
  • 11
1
vote
1 answer

Why my code runs so much slower with joblib.Parallel() than without?

I am new to use joblib.Parallel() to speed up some massive numpy.fft calculations. I follow this example presented on joblib-web Using the example, I can see following result on my computer: Elapsed time computing the average of couple of slices…
Wu-O
  • 21
  • 4
1
vote
0 answers

how to put numpy array entirely on RAM using numpy memmap?

I would like to use a memmap allocated numpy array that can be processed in parallel using joblib i.e. shared memory between different processes. But I also want the big array to be stored entirely on RAM to avoid the write/read to disk that memmap…
danny
  • 1,101
  • 1
  • 12
  • 34
1
vote
0 answers

Is it possible to create a numpy.memmap of array of arrays?

I have a (4,) arrays that I want to save to the disk (The sizes I am working with can not fit into memory so I need to dynamically load what I need). However, I want to have that in a single numpy.memmap. Not sure if it is possible but any…
Kour
  • 33
  • 8
0
votes
0 answers

Pickling memmaped np.ndarray copies array buffer into the pickled object

I'm trying to subclass an np.ndarray with the following constraints: the subclass has to be memory mapped is has to be pickleable while keeping the underlying buffer memory mapped. I was able to implement the subclass, however, when pickling the…
ilcord
  • 314
  • 1
  • 5
  • 15
0
votes
0 answers

Numpy memmap and num_workers: Freezes and restart as executing multiple pytorch scripts

Mostly I work with huge file more than my workstation ram. So I uses numpy memmap file format with pytorch. If I set num_workers more than 1, my workstation freezes and starts to reboot itself. I tried to save those memmap files to each separate…
YilGuk Seo
  • 77
  • 6
0
votes
0 answers

Create a numpy memmap for a large Repeat Python Numpy memmap for binary file

I am trying to construct a memory map file for a large binary file where I give it a dictionary of dtypes, offsets, and shapes and want it to repeat for a specified amount. Is there a way to do this like you can in the Matlab memmapfile.m? …
Michelle
  • 11
  • 3
0
votes
1 answer

how to handle this mmap error in rust : SIGBUS: access to undefined memory?

I am doing a mmap of a file using the crate memmap. The mapping is done like this : let mut mmap_var = memmap::MmapOptions::new().len(8589934592) .map_mut(file).unwrap(); Everything goes well until i reach the value contained in…
Dufy Teguia
  • 97
  • 8
0
votes
0 answers

How Can I disable the MemMap with Davinci Config

I am freshly new with Autosar , I have a MemMap module in my SIP but I want to disable it from Davinci Config when I try to delete it I can't so I thought maybe there are some steps that should be done to disable it, I have looked into the Technical…
baccouri firas
  • 11
  • 2
0
votes
2 answers

Matlab: fastest method of reading parts/sequences of a large binary file

I want to read parts from a large (ca. 11 GB) binary file. The currently working solution is to load the entire file ( raw_data ) with fread(), then crop out pieces of interest ( data ). Question: Is there a faster method of reading small (1-2% of…
user2305193
  • 2,079
  • 18
  • 39
0
votes
0 answers

Change dimension order of numpy shape for image data

I am using numpy and nibabel lib to load data from nii file. This is my code to load the image data import os, glob import nibabel as nib import numpy as np import matplotlib.pyplot as plt DataInputPath = '/content/drive/MyDrive/Data/Volumes/Img'…
jonedabb
  • 59
  • 6
0
votes
1 answer

Numpy memmap negative dimensions not allowed

Numpy memmap Reference page When mode != 'r', even positive offsets beyond end of file are valid; fmt = "IIiiIIddiIiii64sIIII48sxxxx" fmt1 = "?Idd16s" file = open(self.resultName, "wb") file.write(struct.pack(fmt, *self.header)) for i in…
Ivan Timofeev
  • 11
  • 2
1
2