Getting an empty CSV when downloading from URL in Cpp

Question

My goal is to get a CSV or XLS from a specific URL using Cpp.

When opening the following link

http://www.centrodeinformacao.ren.pt/userControls/GetExcel.aspx?T=CRG&P=01-01-2007&variation=PT

, can see in Browser tools

a 302 redirect and the file being actually downloaded from the following URL

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=02-01-2007&variation=PT

as shown in the next image (Request URL)

If I go to any of the two links manually, the a .xls file downloads just fine so we might as well use the one after redirection.

---

I've decided to move on an use libcurl with Visual Studio 2017 on my W10 machine. The recommended way to include libcurl in a Visual Studio 2017 project is to use vcpkg and that's what I used.

---

1. Install vcpkg

1. Open Git Bash, cd C:/Program Files/ and cloned this repo.

2. Open Command Prompt, cd C:/Program Files/vcpkg, run bootstrap-vcpkg.bat

and after run vcpkg integrate install

---

2. Install libcurl

1. Run vcpkg install curl

---

3. Create a new project

1. Simply create Visual C++ > Windows Desktop > Windows Console Application

to be able to use #include <curl/curl.h> right away

---

4. Current Result

Then, inspired in the following answers

• https://stackoverflow.com/a/6951203/9415908

• https://stackoverflow.com/a/1636415/9415908

And using the following code

#include "pch.h"
#define _CRT_SECURE_NO_WARNINGS
#include <iostream>
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written = fwrite(ptr, size, nmemb, stream);
    return written;
}

void downloadFile(const char* url, const char* fname) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    curl = curl_easy_init();
    if (curl) {
        fp = fopen(fname, "wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        fclose(fp);
    }
}

int main(void) {

    downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.csv");

}

I can see a test.csv in the desired folder but it's an empty file.

Answer 1

Once going to that specific URL, a .xls file downloads. I don't mind getting a XLS instead of a CSV and so changing it to I was able to get the file as expected.

downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.xls");

This is the final code

#include "pch.h"
#define _CRT_SECURE_NO_WARNINGS
#include <iostream>
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written = fwrite(ptr, size, nmemb, stream);
    return written;
}

void downloadFile(const char* url, const char* fname) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    curl = curl_easy_init();
    if (curl) {
        fp = fopen(fname, "wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        fclose(fp);
    }
}

int main(void) {

    downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.xls");

}

Answer 2

You have to understand how your shell works.

Note: When you use system() it is the shell that executes the command.

Your command:

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT

Your shell is treating the "&" character as the "and" operator. Execute the left and right side of the "and" as commands. So it is treating the above as three commands:

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG
P=01-01-2007
variation=PT

It fails on the second command P=01-01-2007

To fix this you may want to quote the string.

"http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT"

 op.insert(0, 1, '"');
 op.append(1, '"');
 system(op.c_str());