7

I would like to share memory between C++ and Python.

My problem:

  1. I am working with big data sets (up to 6 GB of RAM) in C++. All calculations are done in c++.
  2. Then, I want to "paste" all my results to a Python program. But I can only write my data on disk, and then read that file from Python, which is not efficient.

Is there any way to "map" memory corresponding to C++ variables so that I may access the data from Python? I don't want to copy 6GB of data onto a hard drive.

Charles
  • 1,384
  • 11
  • 18
PatrykB
  • 1,579
  • 1
  • 15
  • 24
  • 1
    You may want to search Google/Bing for "InterProcess Communication" (IPC); and you may want to see [this](http://stackoverflow.com/questions/6915191/simple-ipc-between-c-and-python-cross-platform) Stackoverflow post – WhiZTiM Oct 16 '16 at 12:29
  • Depending on how large a code base you have in C++, it might be an idea to use something like [Cython](http://cython.org/). – Bendik Oct 16 '16 at 12:41
  • @WhiZTiM thx, currently i am thinking about IPC method: Shared memory, leater about sockets or mapping file to memory. I know that python dont have pointers, but is there any way to access memory via address ? – PatrykB Oct 16 '16 at 12:41
  • @Bendik about 7k of c++ lines, and it is still growing. atm i am thinking about reading data in python via memory adress – PatrykB Oct 16 '16 at 12:43
  • Do you have two programs or one c++-Library called from Python? – Daniel Oct 16 '16 at 12:44
  • @Daniel two programs. one c++ and one in python. I am calling python program from c++ via `system("python myPyProg.py _args_");` – PatrykB Oct 16 '16 at 12:47
  • 2
    You might want to [embed the python interpreter](https://docs.python.org/3/extending/embedding.html) in your c++ program, that would allow you to give python code access to its data using [python APIs](https://docs.python.org/3/c-api/). – spectras Oct 16 '16 at 13:02
  • @spectras thank you. i think its the best idea yet. – PatrykB Oct 16 '16 at 13:26

1 Answers1

4

First path: I think the more appropriate way for you to go is ctypes. You can create a shared library, and then load the functions of the shared library in Python, and fill all the data containers you want in Python.

In Windows, you can create a DLL, and in Linux you can create a shared .so library.

Now this has the advantage that this will be independent of your Python version.

Second path: I think it's less appropriate but you can get it to work, which is the Python C Extension. With this, you can call Python data containers (PyObjects) and fill them inside C.

However, the code you compile here will always need to be linked to Python libraries.

Which one to use?:

  • Use ctypes if you have some functions you want to call in C/C++, and then do the rest of the work in Python.
  • Use Python C Extension if you have some functions you want to call in Python, and you want to do the rest in C/C++.

With both options, you can transfer huge blocks of memory between C++ and Python without necessarily involving any disk read/write operations.

Hope this helps.

Community
  • 1
  • 1
The Quantum Physicist
  • 24,987
  • 19
  • 103
  • 189