1

Right now I have an api that gets many results (800k) and it should parse and stream them as CSV. The (kind of) working code I have is this 

            var query = context.address.AsNoTracking();

            MemoryStream memoryStream = new MemoryStream();
            StreamWriter writer = new StreamWriter(memoryStream);

              foreach (var row in query)
            {
                writer.WriteLine(row.Name+","+row.Surname+","+row.BirthDate);

            }

            writer.Flush();                                 
            memoryStream.Seek(0, SeekOrigin.Begin);
            return new FileStreamResult(memoryStream, "application/octet-stream")
            {
                FileDownloadName = "DataFile.csv"
            };

However this seems to put everything into memory, and store it using up 300 mb of memory. As i understand it, that happens in the foreach statement. Is there a way that i write the line, stream it, and then dispose of it from memory? My goal is to achieve something similar to this without using so much memory

I have also tried it this way, which works for one call, but if multiple api calls are made at the same time, there is weird behaviour

 public async Task makeDif()
    {




            //var query = context.Adresar.AsNoTracking();
        var query = context.Adresar.Select(x => string.Join(",", x.Ime, x.Prezime, x.Datarag + Environment.NewLine)).AsNoTracking();
        Response.ContentType = "application/json";
        Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes("ime,prezime,datarag" + Environment.NewLine)); 
        foreach (var row in query)
                {
               await Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes(row));

            }}
Guillaume S.
  • 1,515
  • 1
  • 8
  • 21
wonder
  • 75
  • 8
  • Is it acceptable for you to use a temporary file to buffer the results ? So first write to file, instead of memory stream, and the trasnmit using `Response.TransmitFile(filePath)` – Pac0 Feb 04 '20 at 09:30
  • I suppose that could be a workaround, but still wouldn't completely solve my . I tried doing this with yield return, but that doesn't return a full csv file, just many lines. – wonder Feb 04 '20 at 09:34
  • 1
    The problem is `MemoryStream`, not `FileStreamResult`. The code is caching everything in memory before returning it. A quick&easy fix would be to export the data to a temporary text file and return that file – Panagiotis Kanavos Feb 04 '20 at 09:57

1 Answers1

3

You can write data directly to response body without need of putting it into a MemoryStream.

Asker edit: This is what I did 

//var query = context.Adresar.AsNoTracking();
var query = context.Adresar.Select(x => string.Join(",", x.Ime, x.Prezime, x.Datarag + Environment.NewLine)).AsNoTracking();
Response.ContentType = "application/json";
Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes("ime,prezime,datarag" + Environment.NewLine)); 
foreach (var row in query)
{
    await Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes(row));
}

and it seems to work fine

Steven He
  • 168
  • 6
  • is that with return Ok(query) ?, because i've tried that. – wonder Feb 04 '20 at 09:40
  • 1
    Response.ContentType = "application/json"; Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes("ime,prezime,datarag" + Environment.NewLine)); foreach (var row in query) { Response.BodyWriter.WriteAsync(Encoding.UTF8.GetBytes(row)); } tried it like this, works fine for one request, but for multiple requests it seems to return More and more data, One request returns 30mb (as it should), yet multiple request return gigabytes of data. Does this have to do with the method being async? – wonder Feb 04 '20 at 09:52
  • you should "await" the write async, IMO. But even though, I'm not sure if the async should be the cause of the explosion of result size. – Pac0 Feb 04 '20 at 10:34
  • As said [here](https://www.reddit.com/r/dotnet/comments/dcuwio/core_3_writing_text_to_responsebody_is_pretty/f2dw4nt?utm_source=share&utm_medium=web2x) by ben_a_adams.. you could use `Response.WriteAsync(row)` passing string directly if you import `using Microsoft.AspNetCore.Http;` – Hugo Freitas May 06 '20 at 14:47