How to download big files using the http module of FSharp.Data?

Question

The following code snippet is found in the FSharp.Data website http://fsharp.github.io/FSharp.Data/library/Http.html. The type of Text and Binary are string and byte[] respectively. It's not good to get the whole 2GB file in memory and then save it to a file.

let logoUrl = "https://raw.github.com/fsharp/FSharp.Data/master/misc/logo.png"
match Http.Request(logoUrl).Body with
| Text text -> 
    printfn "Got text content: %s" text
| Binary bytes -> 
    printfn "Got %d bytes of binary content" bytes.Length

`Http.RequestStream(logoUrl).ResponseStream.CopyTo(outStream)` seems like it would do it. — Daniel, Jul 21 '14 at 18:14
Nice, dose it need to check if the returned `Response` object of `RequestStream` and `ResponseStream` are not null? — ca9163d9, Jul 22 '14 at 14:46
Considering that this is an F#-specific library, I would guess no. — Daniel, Jul 22 '14 at 15:17

score 2 · Accepted Answer · answered Nov 04 '15 at 12:22

I do not think you can keep the same code as on the FSharp.Data website to download huge files. What I use to download large files is

async {
    let! request = Http.AsyncRequestStream(logoUrl)
    use outputFile = new System.IO.FileStream(fileName,System.IO.FileMode.Create)
    do! request.ResponseStream.CopyToAsync( outputFile ) |> Async.AwaitTaskVoid 
} |> Async.RunSynchronously

If you want to try to download infinite file check the complete source (run on your own risk, it is using the The Infinite File Download)

AwaitTaskVoid is here http://stackoverflow.com/questions/8022909/how-to-async-awaittask-on-plain-task-not-taskt — Dzmitry Lahoda, Dec 03 '15 at 13:35

How to download big files using the http module of FSharp.Data?

1 Answers1