Entity Framework - OutOfMemory Exception

Question

I'm developing a Silverlight Business Application and want to implement a "multipart" upload, which splits a single file into parts with a size of 4096KB. To upload these parts from client to server, I'm using a WebClient (client side) and a generic handler (*.ashx, server side).

Strategy: With the first part a new instance of an Entity Framework class is created. This object has a field/property "binary" (in SQL it's a varbinary(MAX) and in Entity Framework it's a byte[]). I store the first part in the property "binary" and execute SaveChanges(). Then, the handler returns the ID (primary key) of this new object to the client.

The second request to the server contains, beside the second part of my file, the ID returned after the first request. On the server, I load the previously created object from the database and append the second part.

myobject.binary = myobject.binary.Concat(bytes).ToArray<byte>();

myobject is the previously created object, bytes the part I want to append to the binary property.

I repeat this "strategy" until the whole file is uploaded to the server. This works fine for files with a maximum size of ~78MB. For files with a size of ~83MB it works sporadic. Files with a size of ~140MB will abort with a OutOfMemory Exception at SaveChanges().

StackTrace

at System.Object.MemberwiseClone()
at System.Array.Clone()
at System.Data.Common.CommandTrees.DbConstantExpression..ctor(TypeUsage resultType, Object value)
at System.Data.Mapping.Update.Internal.UpdateCompiler.GenerateValueExpression(EdmProperty property, PropagatorResult value)
at System.Data.Mapping.Update.Internal.UpdateCompiler.BuildSetClauses(DbExpressionBinding target, PropagatorResult row, PropagatorResult originalRow, TableChangeProcessor processor, Boolean insertMode, Dictionary`2& outputIdentifiers, DbExpression& returning, Boolean& rowMustBeTouched)
at System.Data.Mapping.Update.Internal.UpdateCompiler.BuildUpdateCommand(PropagatorResult oldRow, PropagatorResult newRow, TableChangeProcessor processor)
at System.Data.Mapping.Update.Internal.TableChangeProcessor.CompileCommands(ChangeNode changeNode, UpdateCompiler compiler)
at System.Data.Mapping.Update.Internal.UpdateTranslator.<ProduceDynamicCommands>d__0.MoveNext()
at System.Linq.Enumerable.<ConcatIterator>d__71`1.MoveNext()
at System.Data.Mapping.Update.Internal.UpdateCommandOrderer..ctor(IEnumerable`1 commands, UpdateTranslator translator)
at System.Data.Mapping.Update.Internal.UpdateTranslator.ProduceCommands()
at System.Data.Mapping.Update.Internal.UpdateTranslator.Update(IEntityStateManager stateManager, IEntityAdapter adapter)
at System.Data.EntityClient.EntityAdapter.Update(IEntityStateManager entityCache)
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at MyObjectContext.SaveChanges(SaveOptions options) in PathToMyEntityModel.cs:Line 83.
at System.Data.Objects.ObjectContext.SaveChanges()
at MultipartUpload.ProcessRequest(HttpContext context) in PathToGenericHandler.ashx.cs:Line 73.

Does anyone has an idea, what's wrong with my implementation? If you need more information or code snippets, please let me know it.

Kind regards, Chris

Answer 1

Think about it. After having uploaded (for example) 130 MB, how much memory is required to execute this line:

myobject.binary = myobject.binary.Concat(bytes).ToArray<byte>();

Obviously, the previous array is in memory, that's 130 MB. And somehow the new array must be in memory too, that another 130 MB, right?

It is actually a lot worse. Concat() is producing a sequence, and ToArray() doesn't know how big it will be.

So what .ToArray() does, is that it creates an internal buffer and starts filling it with the output from the .Concat() iterator. Obviously, it does not know how big the buffer should be, so every once in a while it will find that there are more bytes comming in than its buffer can hold. It then needs to create a bigger buffer. What it will do is create a buffer that is twice as big as the previous one, copy everyting over and start using the new buffer. But that means that at some point, the old buffer and the new one must be in memory at the same time.

At some point, the old buffer will be 128 MB, and the new buffer will be 256 MB. Together with the 130 MB of the old file, that is about half a gigabyte. Now let's hope no two (or more) users do this at the same time.

I would suggest you use a different mechanism. For example, store your uploaded chuncks in a temporary file on disk. When a new chunck comes in, just append to the file. Only when the upload is completed, do whatever it is that you have to do to the file, e.g. store it in the database.

Also, be aware that the maximum size of an array in .NET is limited by a 31 bit index. So the maximum size for a byte array is 2 GB, no matter how much RAM you have in the system.

Finally: if you're dealing with memory blocks this big, make sure that you are running in a 64 bit process, and at least on .NET 4.5, so you can take advantage of the Large Object Heap Improvements in .NET 4.5. But even that isn't magic as “Out Of Memory” Does Not Refer to Physical Memory.