0

I have a 3 Dimensional List like this:
List<List<List<Vector128<byte>>>> dimensionLIST = new List<List<List<Vector128<byte>>>>();

Now, when I fill this List with the data I use for calculations, it will take up 20 GB of RAM memory.

The calculations that I do take 10 hours. So I have created a solution where I open up 10 instances of the application, where I have divided the work in 10 pieces which will make the calculation to only take 1 hour (I have 24 cores in the computer)

The problem now, is that each instance of the Form Application will need to read in this dimensionLIST which means that this would take up 20 GB * 10 = 200 GB RAM which is just not a solution.

My question is, if there is any possibility to put this dimensionLIST in some kind of System Wide Global List variable, - which can be accessed by all 10 instances?

(To mention is that I can't divide this dimensionLIST in 10 pieces because each instance actually needs the whole list)

Thank you!

Andreas
  • 1,121
  • 4
  • 17
  • 34
  • 1
    Memory Mapped file, maybe? Or other Shared Memory, File Mapping semantics? Here's something to get you going: https://learn.microsoft.com/en-us/windows/win32/memory/file-mapping?redirectedfrom=MSDN – Andy Aug 04 '20 at 21:42
  • Yes maybe. I have heard about the Memory Mapped file but this is really new to me. I am not sure how that works and if I can put a `List>>>` in such memories? – Andreas Aug 04 '20 at 21:44
  • You would have to re-architect how you do things, for sure... but you are going from 200GB of memory down to 20GB of memory... that kind of magic never comes free. – Andy Aug 04 '20 at 21:46
  • 1
    can't you make your application multi threaded? so the 10 work peaces can be done by one application, but simultanious by different cores. – RomCoo Aug 04 '20 at 21:47
  • 1
    @RomCoo -- that's a good idea. More info here: https://stackoverflow.com/questions/2510593/how-can-i-set-processor-affinity-in-net – Andy Aug 04 '20 at 21:48
  • @RomCoo, I know I have done tests before by creating `new Threads` and it seems that one only gain about 3-4 times the speed in one instance of an application by creating 3-4 new threads as It seems.One needs open new instances to really gain more than 3-4 times the speed. – Andreas Aug 04 '20 at 21:49
  • The memory mapped file seems interesting. I red the link there. Thank you. But I am not sure how it works, if it is possible to put data into a `List>>>` because the data must be held in exactly that type of List? – Andreas Aug 04 '20 at 21:51
  • That looks interesting about the multi threading. I red the link and could see: `threads[0].IdealProcessor = 0;` Now I am not sure, but is it really possible to divide on 10 threads(cores) and actually gain about 10 times the speed as I do when I open up 10 instances of an application? If that is possible then this solution would be the best. – Andreas Aug 04 '20 at 21:58

1 Answers1

1

You can use Named Pipes.

A named pipe is a named, one-way or duplex pipe for communication between the pipe server and one or more pipe clients. All instances of a named pipe share the same pipe name, but each instance has its own buffers and handles, and provides a separate conduit for client/server communication. The use of instances enables multiple pipe clients to use the same named pipe simultaneously. Any process can access named pipes, subject to security checks, making named pipes an easy form of communication between related or unrelated processes.

ALTERNATIVELY

Use a Windows File Mapping Object which allows you to share memory between processes.

As mentioned in a comment Memory Mapped Files could be a viable option