fastest way to fetch value from website in c

Question

I want to make a http request, get the website content and check one specific element. Currently I'm doing it with lcurl (got explanations from a tutorial). Here is an excerpt of my code:

curl_global_init(CURL_GLOBAL_ALL);
CURL *myHandle;
CURLcode result;
struct BufferStruct output;
output.buffer = NULL;
output.size = 0;

//requested value is on this website
char* url = "https://www.somewebsite.com";

//set curl handle to write the website content into buffer
myHandle = curl_easy_init();
curl_easy_setopt(myHandle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(myHandle, CURLOPT_WRITEDATA, (void*)&output);
curl_easy_setopt(myHandle, CURLOPT_URL, url);
curl_easy_perform( myHandle );

//searching for position of the requested value.
uint8_t* value_position = strstr(output.buffer, "value_description");
//copy the value (8 chars) into a buffer defined earlier
strncpy(value, value_position, 8);
//... clean up etc.

The code works. But I wonder if this is the fastest way... Any ideas how to do it faster? Thanks alot!

Welcome to StackExchange. However, you seem to be on the wrong site; for improvement recommendations on basically working code you probably want to go to https://codereview.stackexchange.com. However, if there is a specific problem with a certain part of your code, please explain in more detail and people here will be happy to help. You might want to take the [tour] and read [ask]. When you did that: Welcome to StackOverflow, the Q/A site for specific programmin questions. — Yunnosch, Jun 21 '17 at 05:27
I'm voting to close this question as off-topic because for improvement recommendations on basically working code you probably want to go to codereview.stackexchange.com. — Stargateur, Jun 21 '17 at 05:55
Possible duplicate of [How do you make a HTTP request with C++?](https://stackoverflow.com/questions/1011339/how-do-you-make-a-http-request-with-c) — Chandrahas Aroori, Jun 21 '17 at 06:01
@ChandrahasAroori C, not C++, and more off-topic than duplicate. — Yunnosch, Jun 21 '17 at 06:06
Before going to codereview, make sure to read the following: [Eric Lippert's blog: Which is faster?](https://ericlippert.com/2012/12/17/performance-rant/). It's not 100% targeted at your question, but still relevant. — grek40, Jun 21 '17 at 06:22
Surely that's not the fastest way. You could skip lubcurl and use raw sockets, you could use prepared HTTP requests then you could prepare TCP packets beforehand in order to skip TCP processing in the kernel, you could create specialized custom silicon chip... That way you'd save some 4 or maybe even 5 nanoseconds. But is it worth the time? — el.pescado - нет войне, Jun 21 '17 at 07:05
Does "faster" mean reducing bandwidth? You could try fetching blocks of the web page at a time and perform your search after each block is received. For example, if you find the data you're looking for in the first few blocks, you're done, so you can stop transferring at that point. — Brandin, Jun 21 '17 at 08:18

score 0 · Answer 1 · answered Jun 21 '17 at 08:13

I want to make a http request, get the website content and check one specific element ... But I wonder if this is the fastest way.

First, "fast" in this context is relative as your bottleneck is bandwidth, not CPU. Next, if all you're doing is trying to find a specific element from web content, you could use telnet or if you have ncat installed on your system, you could just do something like the following:

echo "GET /index.php" | ncat www.somewebsite.com 80 | grep "value_description"

This can be scripted and adjusted as needed, but scraping for a single value you can use grep, sed and/or awk to mine the data without the need to build a C application.

Hope that can help.

In grep you can add the -m 1 parameter to stop the search after the first match, and add option e.g. -C 3 to add context lines (e.g. 3 lines) if needed. Stopping after the first match will allow ncat to stop transferring, possibly reducing bandwidth if "value_description" is found near the beginning of the data. — Brandin, Jun 21 '17 at 08:26

fastest way to fetch value from website in c

1 Answers1