libcurl to put stream of data instead of file

Question

We are using libcurl C API in order to send a file over SFTP. This works fine using a code like this :

....
fd = fopen(local_file_full_path.c_str(), "rb");
if (!fd)
{
    cout << "Error opening local file: " << local_file_full_path << endl;
    return -1;
}
curl_easy_setopt(curl_easy_handle, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl_easy_handle, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl_easy_handle, CURLOPT_READDATA, fd);
curl_easy_setopt(curl_easy_handle, CURLOPT_HTTPHEADER, headerList);

However, that means that even though we have our data available - we have to write them into a file, and then pass the file over.

I was wondering if libcurl provides an option in which, we could pass a stream of data, eg a pointer into a struct of our data. If that's the case, is there any example I could use? And does that mean changes are needed on the "receiver" end - as we don't own it.

I don't know if this helps -> https://stackoverflow.com/a/63152986/1737811 — mutantkeyboard, Apr 28 '22 at 14:27
@mutantkeyboard HTTP is not SFTP. That solution won't work in this case, as SFTP doesn't use MIME — Remy Lebeau, Apr 28 '22 at 14:36

Remy Lebeau · Accepted Answer · 2022-04-28T16:06:04.567

You already have 1/2 of the solution - CURLOPT_READDATA . You just need to pair it with a custom callback in CURLOPT_READFUNCTION, then you can pass a pointer to your existing data in CURLOPT_READDATA and have your callback copy that data into libcurl's buffer when it needs the data.

This callback function gets called by libcurl as soon as it needs to read data in order to send it to the peer - like if you ask it to upload or post data to the server. The data area pointed at by the pointer buffer should be filled up with at most size multiplied with nitems number of bytes by your function.

Set the userdata argument with the CURLOPT_READDATA option.

score 0 · Answer 2 · answered Apr 28 '22 at 14:42

If you're on a POSIX system, you can use fmemopen():

SYNOPSIS
#include <stdio.h>

FILE *fmemopen(void *restrict buf, size_t size,
       const char *restrict mode);
DESCRIPTION

The fmemopen() function shall associate the buffer given by the buf and size arguments with a stream. The buf argument shall be either a null pointer or point to a buffer that is at least size bytes long.

Assuming you have a buffer and bytes variables of the appropriate type, your code would look like this:

fd = fmemopen(buffer, bytes, "rb");
if (!fd)
{
    cout << "Error opening local file: " << local_file_full_path << endl;
    return -1;
}
curl_easy_setopt(curl_easy_handle, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl_easy_handle, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl_easy_handle, CURLOPT_READDATA, fd);
curl_easy_setopt(curl_easy_handle, CURLOPT_HTTPHEADER, headerList);

If you're running on Windows, see C - create file in memory. It specifically addresses how to implement the functionality of fmemopen() on Windows.

libcurl to put stream of data instead of file

2 Answers2