0

I have an Ubuntu C++ application that executes shell commands by calling popen(). The popen() call works great usually but sometimes it will fail and will indicate error 12 'Cannot allocate memory'.

I know that the system is not even close to being out of memory, so that's not the problem. I know that the process has roughly 250 files open, which is far less than the OPEN_MAX value of 1024.

What else could be the problem?

I've read elsewhere that I could be at the STREAM_MAX limit (which is 16), but I don't know how to check whether or not this is actually the case. Any help would be very much appreciated!

Here is the complete code:

// Format the command to get the number of files open in this process
char szCommand[1024];
memset(&(szCommand[0]), 0, sizeof(szCommand));
snprintf(szCommand, sizeof(szCommand) - 1, "lsof -p %i | wc -l", static_cast<int>(getpid()));

// A string to capture the output
std::string szTotalOutput("");

// Execute the command
FILE *pFile = popen(szCommand, "r");

// If the open was successful
if (pFile)
{
    // A text buffer to capture output
    char szOutput[2048];
    memset(&(szOutput[0]), 0, sizeof(szOutput));

    // Get the output
    while (fgets(szOutput, static_cast<int>(sizeof(szOutput) - 1), pFile))
    {
        // Add to the total output
        szTotalOutput += std::string(szOutput);
    }

    // Close the file
    pclose(pFile);
}

// Or, if the open failed
else
{
    // An error string
    char szError[2048];
    memset(&(szError[0]), 0, sizeof(szError));

    // If we have a good error
    if (errno != 0)
    {
        // Format the error string
        snprintf(szError, sizeof(szError) - 1, "popen failed with error %i '%s'", errno, strerror(errno));
    }

    // Or, if we don't have a good error
    else
    {
        // Format the error string
        snprintf(szError, sizeof(szError) - 1, "popen failed but errno was not set");
    }

    // Print this error message to the console window
    printf("Error: File %s line %i: %s\n", __FILE__, __LINE__, szError);
}

EDIT:

I've found that if I re-enable swap (which was previously disabled) then the issue doesn't happen. But ultimately I need for this to work with swap disabled.

user2062604
  • 247
  • 3
  • 16
  • Hi did you try/refer the below posting ? [Here][1] [1]: http://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory – GManika Jun 24 '15 at 18:15
  • Do you call `pclose` for each call to `popen`? Maybe you call `fclose` by mistake. – David Schwartz Jun 24 '15 at 18:20
  • @user2062604 Just to be clear, is your call to `popen` failing and leaving `errno` set to `ENOMEM`, or could this be happening in one of the programs created by your call to `popen`? Note that `popen("lsof -p pid | wc -l", "r")` should fork a shell which will then fork for `lsof` and again for `wc`. – nategoose Jun 24 '15 at 19:16
  • @DavidSchwartz Thanks for checking, but yes I am calling pclose – user2062604 Jun 24 '15 at 20:37
  • @AndrewHenle I've edited my question to include the complete code. – user2062604 Jun 24 '15 at 20:37
  • So the full message printed by your program is "popen failed with error 12 'Cannot allocate memory'"? – nategoose Jun 24 '15 at 20:43
  • @nategoose I think it really is popen that is setting errno because I set errno to zero before calling popen and I doubt that lsof or wc are causing ENOMEM. Could I be wrong here? – user2062604 Jun 24 '15 at 20:53
  • @nategoose The full message is: Error: File ShellCommand.cpp line 137: popen failed with error 12 'Cannot allocate memory' – user2062604 Jun 24 '15 at 20:53
  • I'm particularly interested in the STREAM_MAX limit, how can I tell if I'm at or close to this limit? Am I barking up the wrong tree? – user2062604 Jun 24 '15 at 20:58
  • @user2062604 I was just wanting to verify that the error message wasn't coming from one of the programs that your program spawned. – nategoose Jun 24 '15 at 20:58
  • Sounds like a fork failure because your process is so big there isn't room for a copy of it. Related: [this question](http://stackoverflow.com/q/8368067/2404501) –  Jun 24 '15 at 21:01
  • @WumpusQ.Wumbley `fork` isn't that memory hungry, though `exec` can be based on the size of the executable. `pipe` can be as well, and there are 2 pipes involved here. – nategoose Jun 24 '15 at 21:16
  • @user2062604 Temporarily replace your `popen` code with `system("szCommand")` and see what happens. You'll have to handle return code checking differently. If that works then try putting `system("cat /proc/meminfo");` just before your call to `popen` and see what the output is. – nategoose Jun 24 '15 at 21:20
  • @nategoose I've tried that as well, but when system(szCommand) fails, it just returns -1. Also, I'm capturing data from /proc/meminfo every second and logging the memory used, as well as watching the Ubuntu graphical system monitor, and the system memory used never goes over 60%. – user2062604 Jun 24 '15 at 21:38
  • @user2062604 This is a rather simple program to "need for this to work with swap disabled". Can you elaborate on what the requirement is? If you're hitting the memory limit when running this then you've got some problems. You should be able to run it in console mode (with no GUI) with swap disabled though. You could also call the system calls directly and avoid `popen` opening a shell and decreasing buffer sizes, which would save you a little memory, but not that much. – nategoose Jun 25 '15 at 02:43
  • @nategoose This is a small part of a much larger program. The issue here is that I'm not, as far as I can tell, hitting the memory limit. Both /proc/meminfo and the graphical System Monitor both indicate that I'm nowhere close to the memory limit. I must be hitting some other resource limit, but I don't know what it is. I have a hunch that it might be STREAM_MAX but I don't know how to check for this. If by "run the system calls" you mean call system() then I have already tried that and it fails, returning -1, just as frequently. What could be going wrong here? Thanks for everyone's help btw! – user2062604 Jun 25 '15 at 03:40
  • @user2062604 When I said "system calls" I meant `fork()`, `pipe()`, `execve()`, `dup()`, `dup2()`, and `sigprocmask()`. Those are called by `popen()`, but in `popen` your shell is run first and it runs the command string, which is useful since it looks up the locations of the commands with the PATH environmental variable and can also do built in commands that aren't separate programs, but calling them directly has less overhead or tell you more about what exactly is failing. – nategoose Jun 25 '15 at 13:19
  • I re-implemented popen() as @nategoose suggested and turns out fork() is failing with ENOMEM. I still believe this is because I have some resource leak - this is backed up by the fact that my system got to a point where I could hardly run my application without fork() failing, even after restarting the application, but after rebooting, fork() stopped failing entirely (at least for the time being). The problem is I have no idea what resource limit I'm hitting - it doesn't seem to be memory (which is only at 60% capacity). Does anyone know what other limits I could be hitting and how to test? – user2062604 Jun 26 '15 at 17:03
  • @user2062604 Have you looked in the logs? There may be some messages in there relevant to the failing call to `fork`. http://www.cyberciti.biz/faq/linux-log-files-location-and-how-do-i-view-logs-files/ – nategoose Jun 29 '15 at 03:41
  • Thanks for the good idea @nategoose but I did a text search through the entire /var/log directory for "fork" as well as "13:28:19" which was the system time when I could last get the error to occur, and nothing useful came up. – user2062604 Jun 29 '15 at 18:39

0 Answers0