binary serialization/deserialization (BinaryFormatter) in Parallel.ForEach

Question

Perhaps this is a duplication(( But I have not found any sample code of the using of binnaryformatter in Parallel.Foreach. Could anyone provide a sample of it? Sample of code

Parralel.ForEach(files, fileCurr=>
                  {
                      using(lib.Accesser("fileType", fileNameSpec))
                      {
                        LoadFileData(fileNameSpec,fileCurr,cancellationToken, progressCallback);
                      }
                   });
and

LoadFileData(fileNameSpec,fileCurr,cancellationToken, progressCallback)
{
 using(lib2.load(fileCurr.name))
{
 foreach(var v in fileCurr.include)
 {
   var objectForSerialization = loadObj(v);
   //my code goes below
   System.IO.Stream stream = System.IO.File.Open("J:\\volume", FileMode.OpenOrCreate);
                    System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bformatter = new BinaryFormatter();
                    bformatter.Serialize(stream, objectForSerialization);
                    stream.Close();
  //but this fil will be locked by multyply writes

 }
}
}

Is there any reason why your serialisation and/or deserialisation would be problematic in parallel processing? Does it depend on properties of other objects that are in the same processing queue for instance? What have you tried? Can we see some example code? — Timothy Groote, Feb 21 '13 at 10:33
where is the problem ? Any exception ? Just put the serialisation code in a Methode like serialize(xx) and call thius from the parallel forearch. Just make sure that they dont share any ressources — Boas Enkler, Feb 21 '13 at 10:47
multyply writes in file(( I need a simple sample how to sincronize reads and writes — curiousity, Feb 21 '13 at 10:48

score 1 · Answer 1 · edited May 23 '17 at 10:32

1

One part of your problem is that

reading and writing or executing multiple writes in the same file in parallel is not realistically possible.

Once you implement a locking mechanism for the file reads / writes, there is no longer any need to execute your iteration in parallel because its operations cannot be executed in parallel.

If you are only deserialising, (just reading from the files) it's a different story.

Read-only access can be done in parallel because it does not require locking of the file per sé.

see : How can I read a text file without locking it?

Another thing you could do, is preload the text files (if this is possible) and deserialise them straight from the resulting strings.

::edit::

one more thing :

You appear to always be writing to the same file on your J: drive (a file called volume with no extention).

If you are serialising multiple objects, you will want them to be serialised to different files...

If you can assure yourself that all objects are serialised to different files, you can make the operation parallel.

This would be someting like this:

 System.IO.Stream stream = System.IO.File.Open("J:\\volume\\" + GenerateFileName(v), FileMode.OpenOrCreate);

Then make sure GenerateFileName generates a unique file name as string.

edited May 23 '17 at 10:32

Community

1
1

answered Feb 21 '13 at 10:59

Timothy Groote

8,614
26
52

Thank you for the clear answer, Mr. Timothy. The problem is that this object "objectForSerialization" creates only in this place(( and it is not passed to other functions. How dou you think - is it posible to serialize this obects in my situation? – curiousity Feb 21 '13 at 11:03
That's hard for me to tell, since i can't see what type `objectForSerialization` is, or what exactly happens in `loadObj(v)` – Timothy Groote Feb 21 '13 at 11:08
i did notice one more thing, please look at the updated answer – Timothy Groote Feb 21 '13 at 11:09
Oh, and just "Tim" is fine :P – Timothy Groote Feb 21 '13 at 11:14
I am new in this project so I am only making my first steps but - objectForSerialization is a really LARGE class with a number of guid, string and so on fields - that is ok. For fields like ManualResetEvent or SemaphoreSlim I use a [NonSerialized()]. The loadObj(v) - is simply a "constructor" as I understand - It creates new instance of this objectForSerialization and sets a fields of it with values from incoming "v", some values initialized by default values as I understood. But I am only trying to serialize in binary file this object – curiousity Feb 21 '13 at 11:15

binary serialization/deserialization (BinaryFormatter) in Parallel.ForEach

1 Answers1