MonoTouch - WebRequest memory leak and crash?

Question

I've got a MonoTouch app that does an HTTP POST with a 3.5MB file, and it is very unstable on the primary platforms that I test on (iPhone 3G with OS 3.1.2 and iPhone 4 with OS 4.2.1). I'll describe what I'm doing here and maybe someone can tell me if I'm doing something wrong.

In order to rule out the rest of my app, I've whittled this down to a tiny sample app. The app is an iPhone OpenGL Project and it does only this:

1. At startup, allocate 6MB of memory in 30k chunks. This simulates my app's memory usage.
2. Read a 3.5MB file into memory.
3. Create a thread to post the data. (Make a WebRequest object, use GetRequestStream(), and write the 3.5MB data in).
4. When the main thread detects that the posting thread is done, goto step 2 and repeat.

Also, each frame, I allocate 0-100k to simulate the app doing something. I don't keep any references to this data so it should be getting garbage collected.

iPhone 3G Result: The app gets through 6 to 8 uploads and then the OS kills it. There is no crash log, but there is a LowMemory log showing that the app was jettisoned.

iPhone 4 Result: It gets an Mprotect error around the 11th upload.

A few data points:

• Instruments does NOT show the memory increasing as the app continues to upload.
• Instruments doesn't show any significant leaks (maybe 1 kilobyte total).
• It doesn't matter whether I write the post data in 64k chunks or all at once with one Stream.Write() call.
• It doesn't matter whether I wait for a response (HttpWebRequest.HaveResponse) or not before starting the next upload.
• It doesn't matter if the POST data is even valid. I've tried using valid POST data and I've tried sending 3MB of zeros.
• If the app is not allocating any data each frame, then it takes longer to run out of memory (but as mentioned before, the memory that I'm allocating each frame is not referenced after the frame it was allocated on, so it should be scooped up by the GC).

If nobody has any ideas, I'll file a bug with Novell, but I wanted to see if I'm doing something wrong here first.

If anyone wants the full sample app, I can provide it, but I've pasted the contents of my EAGLView.cs below.

using System;
using System.Net;
using System.Threading;
using System.Collections.Generic;
using System.IO;
using OpenTK.Platform.iPhoneOS;
using MonoTouch.CoreAnimation;
using OpenTK;
using OpenTK.Graphics.ES11;
using MonoTouch.Foundation;
using MonoTouch.ObjCRuntime;
using MonoTouch.OpenGLES;

namespace CrashTest
{
    public partial class EAGLView : iPhoneOSGameView
    {
        [Export("layerClass")]
        static Class LayerClass ()
        {
            return iPhoneOSGameView.GetLayerClass ();
        }

        [Export("initWithCoder:")]
        public EAGLView (NSCoder coder) : base(coder)
        {
            LayerRetainsBacking = false;
            LayerColorFormat = EAGLColorFormat.RGBA8;
            ContextRenderingApi = EAGLRenderingAPI.OpenGLES1;
        }

        protected override void ConfigureLayer (CAEAGLLayer eaglLayer)
        {
            eaglLayer.Opaque = true;
        }


        protected override void OnRenderFrame (FrameEventArgs e)
        {
            SimulateAppAllocations();
            UpdatePost();           

            base.OnRenderFrame (e);
            float[] squareVertices = { -0.5f, -0.5f, 0.5f, -0.5f, -0.5f, 0.5f, 0.5f, 0.5f };
            byte[] squareColors = { 255, 255, 0, 255, 0, 255, 255, 255, 0, 0,
            0, 0, 255, 0, 255, 255 };

            MakeCurrent ();
            GL.Viewport (0, 0, Size.Width, Size.Height);

            GL.MatrixMode (All.Projection);
            GL.LoadIdentity ();
            GL.Ortho (-1.0f, 1.0f, -1.5f, 1.5f, -1.0f, 1.0f);
            GL.MatrixMode (All.Modelview);
            GL.Rotate (3.0f, 0.0f, 0.0f, 1.0f);

            GL.ClearColor (0.5f, 0.5f, 0.5f, 1.0f);
            GL.Clear ((uint)All.ColorBufferBit);

            GL.VertexPointer (2, All.Float, 0, squareVertices);
            GL.EnableClientState (All.VertexArray);
            GL.ColorPointer (4, All.UnsignedByte, 0, squareColors);
            GL.EnableClientState (All.ColorArray);

            GL.DrawArrays (All.TriangleStrip, 0, 4);

            SwapBuffers ();
        }


        AsyncHttpPost m_Post;
        int m_nPosts = 1;

        byte[] LoadPostData()
        {
            // Just return 3MB of zeros. It doesn't matter whether this is valid POST data or not.
            return new byte[1024 * 1024 * 3];
        }

        void UpdatePost()
        {
            if ( m_Post == null || m_Post.PostStatus != AsyncHttpPostStatus.InProgress )
            {
                System.Console.WriteLine( string.Format( "Starting post {0}", m_nPosts++ ) );

                byte [] postData = LoadPostData();

                m_Post = new AsyncHttpPost( 
                    "https://api-video.facebook.com/restserver.php", 
                    "multipart/form-data; boundary=" + "8cdbcdf18ab6640",
                    postData );
            }
        }

        Random m_Random = new Random(0);
        List< byte [] > m_Allocations;

        List< byte[] > m_InitialAllocations;

        void SimulateAppAllocations()
        {
            // First time through, allocate a bunch of data that the app would allocate.
            if ( m_InitialAllocations == null )
            {
                m_InitialAllocations = new List<byte[]>();
                int nInitialBytes = 6 * 1024 * 1024;
                int nBlockSize = 30000;
                for ( int nCurBytes = 0; nCurBytes < nInitialBytes; nCurBytes += nBlockSize )
                {
                    m_InitialAllocations.Add( new byte[nBlockSize] );
                }
            }

            m_Allocations = new List<byte[]>();
            for ( int i=0; i < 10; i++ )
            {
                int nAllocationSize = m_Random.Next( 10000 ) + 10;
                m_Allocations.Add( new byte[nAllocationSize] );
            }
        }       
    }




    public enum AsyncHttpPostStatus
    {
        InProgress,
        Success,
        Fail
    }

    public class AsyncHttpPost
    {
        public AsyncHttpPost( string sURL, string sContentType, byte [] postData )
        {
            m_PostData = postData;
            m_PostStatus = AsyncHttpPostStatus.InProgress;
            m_sContentType = sContentType;
            m_sURL = sURL;

            //UploadThread();
            m_UploadThread = new Thread( new ThreadStart( UploadThread ) );
            m_UploadThread.Start();            
        }

        void UploadThread()
        {
            using ( MonoTouch.Foundation.NSAutoreleasePool pool = new MonoTouch.Foundation.NSAutoreleasePool() )
            {
                try
                {
                    HttpWebRequest request = WebRequest.Create( m_sURL ) as HttpWebRequest;
                    request.Method = "POST";
                    request.ContentType = m_sContentType;
                    request.ContentLength = m_PostData.Length;

                    // Write the post data.
                    using ( Stream stream = request.GetRequestStream() )
                    {
                        stream.Write( m_PostData, 0, m_PostData.Length );
                        stream.Close();
                    }

                    System.Console.WriteLine( "Finished!" );

                    // We're done with the data now. Let it be garbage collected.
                    m_PostData = null;

                    // Finished!
                    m_PostStatus = AsyncHttpPostStatus.Success;
                }
                catch ( System.Exception e )
                {
                    System.Console.WriteLine( "Error in AsyncHttpPost.UploadThread:\n" + e.Message );
                    m_PostStatus = AsyncHttpPostStatus.Fail;
                }
            }
        }

        public AsyncHttpPostStatus PostStatus
        {
            get
            {
                return m_PostStatus;
            }
        }


        Thread m_UploadThread;

        // Queued to be handled in the main thread.
        byte [] m_PostData;

        AsyncHttpPostStatus m_PostStatus;
        string m_sContentType;
        string m_sURL;
    }
}

Answer 1

I think you should read in your file 1 KB (or some arbitrary size) at a time and write it to the web request.

Code similar to this:

byte[] buffer = new buffer[1024];
int bytesRead = 0;
using (FileStream fileStream = File.OpenRead("YourFile.txt"))
{
    while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
    {
        httpPostStream.Write(buffer, 0, bytesRead);
    }
}

This is off the top of my head, but I think it's right.

This way you don't have an extra 3MB floating around in memory when you don't really need to. I think tricks like this are even more important on iDevices (or other devices) than on the desktop.

Test the buffer size too, a larger buffer will get you better speeds up to a point (I remember 8KB being pretty good).