How to transfer large class types from REST server to controller

Question

I would like some advice on the best way to transfer custom objects from a "REST" server to a controller. In brief, we are running MVC 4 on a server that handles HttpGet requests. Initially, all responses from the server could be accommodated by serializing our custom classes to JSON and then deserializing them client size. However, now that we need to transfer large data (i.e. images) this approach no longer works.

The followings is condensed versions of what I am currently doing, but I would like some advice on a better way to transfer large files (custom types).

classes:

[Serializable]
public class TradingPost
{
    public int TradingPostId { get; set; }
    public string UserId { get; set; }        
}

[Serializable]
public class TradingPostImage
{
    public int TradingPostImageId { get; set; }
    public string ImageName { get; set; }
    public string ImageData { get; set; }
}

[Serializable]
public class TradingPostWithImages
{
    public TradingPost Post { get; set; }
    public List<TradingPostImage> Images { get; set; }

    public TradingPostWithImages()
    {
        Post = new TradingPost();
        Images = new List<TradingPostImage>();
    }
}

Server side controller:

[HttpGet]
    public ActionResult GetAllTradingPosts()
    {            
        List<TradingPostWithImages> postsAndImages = PostRepository.GetAllPostsAndImages();

        MemoryStream memoryStream = new MemoryStream();
        BinaryFormatter binaryFormatter = new BinaryFormatter();
        binaryFormatter.Serialize(memoryStream, postsAndImages);

        var postsAsByteArray = memoryStream.ToArray();

        return File(postsAsByteArray, "application/octet-stream");              
    }

Client side controller:

public List<TradingPostWithImages> GetItems()
    {
        var dataGatewayPath = GetGateWayPath("GetTradingPosts");

        MemoryStream webData = new MemoryStream();

        var Request = WebRequest.CreateHttp(dataGatewayPath );
        var Response = Request.GetResponse() as HttpWebResponse;

        if(Response != null)
        {
             DataStream = Response.GetResponseStream();

             if(DataStream != null)
             {
                    DataStream.CopyTo(webData);
                    DataStream.Close();
             }

             Response.Close();
        }

        webData.Position = 0;
        BinaryFormatter formatter = new BinaryFormatter();

        var activeItems = (List<TradingPostWithImages>)formatter.Deserialize(webData);

        return activeItems;
    }

This works, but is there a way I can accomplish what I am after without deserializing a stream and then casting it to my custom type? Is there a better, or more efficient way to accomplish my task?

Answer 1

To directly answer your question based on some of the conversation in the comments, really, the only thing I can suggest is that you take a lazy load type approach. Typically, with lazy loading, you request data be loaded as needed, but in this case, all data will be needed right away, but the idea is just the same. So the most efficient way I could suggest is that you load all of the simple data first (basically the data transfers fairly quickly which is pretty much everything but your images in this case). I don't think you're going to get any more efficient than a byte stream over http (maybe try UDP??), so performance can be tuned via user perception.

If your view layer is a web page, then you can just render all of the basic object data first, by referencing image IDs in the thin object on the first call (rather than sending a fat object complete with image data), then use those IDs for image tags that then call a content action which will retrieve the image data from the server asynchronously using that id. This way, your page is usable even while the images load. Like so:

If your view layer is something like a winforms app, you can do roughly the same thing. The idea being that the simple data is retrieved and displayed immediately while images are loaded on a different thread(s), keeping your UI responsive. Just display some sort of loading indicator where the images are about to display.

IMO, when there's large file transfers happening over HTTP, there's only so much you can do to be [creatively] efficient. It's more about dealing with interrupted connections with retry logic (resume download, via byte offset, etc) and keeping the UI responsive than anything else. As long as visible progress is being made without the user feeling like the application is locked up, the negative experience caused by those lengthy load times should be mitigated.

So beyond this messaging strategy, I'd just look into techniques for streaming. This article may help: WebAPI Request Streaming support