1

I'm creating an app to show time tables. In order to do that, I need to download huge amount of data to my API. There's nearly 6000 JSON files to download at the beggining of the month (approximately 190 bus lines and each line have URLs for each day of the month) and amount of JSONs to download decreases daily (on second day it will be ~5800 JSONs, on third day ~5600 and so on) from my local public transport API. Each JSON have about 3 MB and with 6000 URLs at the beggining of each month, it's about 17 GB of data. With that, I was wondering if there's a faster way to do it.

At the beggining I tried to use this function:

public async static Task<List<string>> MassDataDownload(List<string> urlList)
    {
        List<string> listOfJsons = new List<string>();

        using (HttpClient client = new HttpClient())
        {
            foreach(var url in urlList)
            {
                HttpResponseMessage response = await client.GetAsync(url);
                var json = string.Empty;

                if (response.IsSuccessStatusCode)
                    json = await response.Content.ReadAsStringAsync();
                else
                    continue;

                listOfJsons.Add(json);
            }
        }

        return listOfJsons;
    }

but it took almost 10 minutes to download not even 1/10 of links. Then I came across this mass data download SO page and my mistake was to use one HttpClient to download AND I was trying to reach one site lot of times within short amount of times. Based on information there, I created this function:

public async static Task<List<string>> MassDataDownload(List<string> urlList)
    {
        BlockingCollection<HttpClient> ClientQueue = new BlockingCollection<HttpClient>();
        urlList.ForEach(x => ClientQueue.Add(new HttpClient()));
        List<string> listOfJsons = new List<string>();

        foreach (var url in urlList)
        {
            var worker = ClientQueue.Take();
            var json = await worker.GetStringAsync(url);
            worker.Dispose();
            listOfJsons.Add(json);
        }

        return listOfJsons;
    }

but still it's downloading slow. Is there any faster way to download this data or is there any framework that could help me to achieve that?

ProfSJonalista
  • 11
  • 2
  • 2
    this might help: https://stackoverflow.com/questions/8869812/a-faster-way-to-download-multiple-files – Md. Tazbir Ur Rahman Bhuiyan Nov 16 '18 at 12:30
  • 1
    Even if you change your download code its still 17 GB of data. I would be suprised if you won't be rate-limited or blocked if you try to download to fast. Why do you need to download it so quickly? Can you download the data in advance so you already have it locally when the new month begins? – Robert Nov 16 '18 at 12:32
  • One issue with your design is that the contents of all of the JSON from all requests must be in memory at once - consider alternatives to avoid that. You'll also likely want to increase https://learn.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.defaultconnectionlimit?redirectedfrom=MSDN&view=netframework-4.7.2#System_Net_ServicePointManager_DefaultConnectionLimit from its default (2) to a larger value. – mjwills Nov 16 '18 at 12:34
  • 1
    Possible duplicate of [C# Download data from huge list of urls](https://stackoverflow.com/questions/19389938/c-sharp-download-data-from-huge-list-of-urls) – mjwills Nov 16 '18 at 12:37
  • I would caution you against following the first link (suggested by Tazbir). It uses `Parallel.ForEach`. That is not necessary here. Spinning up threads for IO bound work is not optimal. – mjwills Nov 16 '18 at 12:38
  • 1
    [How to limit the amount of concurrent async I/O operations?](https://stackoverflow.com/questions/10806951/how-to-limit-the-amount-of-concurrent-async-i-o-operations?answertab=active#tab-top). Don't focus just on the accepted answer. – Jimi Nov 16 '18 at 13:18
  • Robert, updates are daily and I wanted to update it. Since this is my engineering project, I won't be executed if data in app will be actual or not (I won't have much time to present it) but I wanted my app to look professionally, hence the question. As for other replies, thank you all, I will look into them – ProfSJonalista Nov 16 '18 at 16:06

0 Answers0