2

I use that very simple C program to execute a system call to php each second, in order to run a php script that sends pending push notification in my database to APNS (Apple notification service).

Anyway, this program causes a memory overflow after about 10 hours, so I reduced sleep time between thread creation from 1s to 10000us, and I could see in real time with htop that memory were increasing without never lower. Here is the program :

#include <stdlib.h>
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

typedef struct {
    char* script_path ;
} arg_for_script ;

static void *start_instance(void *_args)
{
    int id = abs(pthread_self());
    arg_for_script* args = _args ;

    printf("[SERVICE] start php script on thread %d\n",id);
    fflush(stdout);
    char cmd[200] ;

    sprintf(cmd, "php -f %s %d", args->script_path, id );

    system(cmd);

    printf("[SERVICE] end of script on thread %d\n", id);
    fflush(stdout);



    pthread_exit(NULL);
}

int main(int argc, char* argv[])
{


    if(argc < 2)
    {
        fprintf(stderr, "[SERVICE] Path of php notification script must be filled\n");
        fflush(stderr);
        return EXIT_FAILURE;
    }

    arg_for_script args ;

    args.script_path = argv[1];

    pthread_attr_t tattr ;
    struct sched_param param;
    param.sched_priority = 1 ;

    pthread_attr_init(&tattr);

    pthread_attr_setinheritsched(&tattr, PTHREAD_EXPLICIT_SCHED);
    pthread_attr_setschedpolicy(&tattr, SCHED_FIFO);
    pthread_attr_setschedparam(&tattr, &param);



    while(1) {
        pthread_t thrd;

      //  if(pthread_create(&thrd, &tattr, start_instance, (void *)&args) == -1) {
        if(pthread_create(&thrd, NULL, start_instance, (void *)&args) == -1)
        {
            fprintf(stderr, "[SERVICE] Unable to create thread\n");
            fflush(stderr);
            return EXIT_FAILURE;
        }

        usleep( 10000);
    }

  //  pthread_attr_destroy(&tattr);
    return EXIT_SUCCESS ;

}

Here, I don't dynamically allocate any RAM with malloc. Why would this program increases memory usage ? What pointer should I free here ?

Jerem Lachkar
  • 1,008
  • 11
  • 24

1 Answers1

4

You aren't calling pthread_join() nor use pthread_detach(), so the resources allocated for the thread aren't freed. Namely each thread has it's own stack, which is probably what causes the rising memory consumption.

Some remarks about your implementation: Since you plan on executing a PHP script with system() and don't actually need to work on shared variables or file descriptors, it's better to use fork() and one of the variants of exec(). This will spawn a new process without the intermediate step of creating a thread. It's also not recommended to use system() because it often allows to exploit the program when the input isn't properly sanitized. In this case it might be fine, if you only call it manually.

  • Thanks that was pthread_detach() :) actually my php script contains : (1) retrieve all pending-marked notification from db and mark them as sent in db (2) send them to Apple via TCP. I want that if any of the php scripts is doing (1), a mutex in my c program blocks any new thread creation (if there are 100K pending notif to send at once, read/write operation in db can take up to 5 minutes.., and if new scripts r called from C in this case, 2 scripts can retrieve same notif and send it twice to Apple). For part (2), it’s ok to let new thread creation. This behavior can be done without thread ? – Jerem Lachkar Jun 05 '19 at 23:06
  • That depends on how you intend to check that case (1) is happening. How would you check that condition when using threads, does the php script print something to stdout? If that's the case you can use exec and use a pipe to redirect the output to your c program (take a look at how to use `dup2`). I'm wondering though, why don't you extend the php script to check condition (1) periodically and poll if new notifications are pending, that would probably be the most straight forward solution. –  Jun 06 '19 at 09:24
  • I wanted to use only a php script executed by cron but cron is not available to launch second-periodic tasks (minutes is the shortest unit of cron). Php script produce output but this is English more intended for developer that want to see what’s going on with journalctl.. I could also produce an extra file containing one bit (1/0) but that’s not a very convinient solution.. I thought about signals, are they intended for that ? I could launch a signal from php to C parent process that continually fork to prevent it from creating any new fork until it receive a « continue » signal ? – Jerem Lachkar Jun 06 '19 at 09:55
  • I feel like you're trying to solve this in a really complicated way. Why don't you extend the php script to run indefinitely and check within the php script once in a while if new data has to be processed. Then you would only need to start the script once to run in the background with something like `nohup php script.php &`. –  Jun 06 '19 at 10:07
  • I wanted to make my push notification able to send a important amount of notification at once, sometimes. Sending like 100K notifications to Apple via TCP can take about 15 min. Meanwhile, new pending notif can come, and I need to send them immediately, so I need to poll db and create new TCP con every single second even if there is another process currently sending a important amount of other notif. That’s why I created this complicated solution but I’m not very used to all this (system programming) so if you have better solution for this flexible purpose I would really appreciate – Jerem Lachkar Jun 06 '19 at 10:20
  • Creating new TCP connections for every push notification shouldn't speed up anything, in fact you're creating a lot of unnecessary round trips. Is there any particular reason why you can't keep a single TCP connection open and send the notification through that one instead of creating a new one every time? –  Jun 06 '19 at 10:30
  • I create a new tcp con for each script, but one script holds all new pending notif that were pushed to db the last second, and mark them as « sending » in db so the following script doesn’t take them to send. Actually that’s true maybe I could keep the same tcp con for each poll, but what I really need is that, if a script discover new 100k notif to send to all users for exemple at 12am (let’s say an ad notif not important), if a user sends a message at 12.01am, this notif much more important must be delivered in real time (I can’t wait that all the ad notif are sent before sending new notif) – Jerem Lachkar Jun 06 '19 at 10:42
  • That’s why I did these parallel tcp con, because I can send big groups of notif to Apple at once, while keeping sending new notification that would occur meanwhile. If I only have one tcp con all notifs are fifo queued and if I send 100k ad notif to Apple, « new message » notif much more important coming after would have to wait until all ad notif are sent.. – Jerem Lachkar Jun 06 '19 at 10:46
  • Ok there are two problem. 1) Currently your performance is bad because you call the script every second and create new TCP connections. Instead modify the script, call it once, keep the TCP connection open and poll the DB frequently for the low priority messages within the script. 2) You need to prioritize notifications. You could either use a datastructure that allows to insert them in front of the message queue of notifs you're trying to send and use the very same TCP connection and polling mechanism or have a separate script only handling the high priority notifs on a separate connection. –  Jun 06 '19 at 10:49
  • Ok the solution of polling while sending and put high priority notif in front of queue looks good, but how can I poll while sending ? Let’s say a poll find 100k notif, it will take time to send them through TCP, and as php is procedural it will block my poll loop. so maybe I need to create one other PHP thread to poll every second and have the notif queue shared between the « sending thread » and the « polling thread » ? – Jerem Lachkar Jun 06 '19 at 11:03
  • I'm not very familiar with PHP it's probably best to post another question for this. You can certainly create an event loop that alternates between sending a single (or small group of) notification(s) and checking the db. If you can write an actual multithreaded PHP script (https://stackoverflow.com/questions/70855/how-can-one-use-multi-threading-in-php-applications) then indeed you could create one thread for polling that fills the queue and on thread for sending that takes notifications from the queue (producer-consumer pattern). –  Jun 06 '19 at 11:09
  • I will try to implement that this afternoon yes :) And create a discution about that on Stackverflow. thank you for all your answer!! – Jerem Lachkar Jun 06 '19 at 11:45