How to pass variable to shell command in C?

Question

For example, I code :

fp = popen("wc -l < myfile", "r");

But myfile should be any file's name which is parsed to this project. It could be file abc.txt or 123.txt or xy.txt etc.

Then I want to get the output of executing this wc -l < myfile. But the problem is that I don't know which function in C can help me to parse the name of the myfile to this shell command and I can also get the output. Can anyone gives me some suggestions?

Edit: The file I want to read is very large. I want to read its data into an array.I cannot use list to store it, because it is too slow to locate a specific data in list. The problem is that if I use one dimensional array to malloc() memory space to the array, there is no enough continuous memory space on the laptop. Therefore, I plan to use two dimensional array to store it. So I have to get the num of lines in the file and then decide the size of each dimensional in this array via log.

Thanks for all answers. This project is about reading two files. The first file is much larger than the second file. The second file is like:

1   13  0
2   414 1
3   10  0
4   223 1
5   2   0

The third num in each line is called "ID". For example, num "1" has ID 0, num "2" has ID 1, num "3" has ID "0". (Ignore the middle num in each line) And the first file is like:

1   1217907
1   1217908
1   1517737
1   2
2   3
2   4
3   5
3   6

If each num in the first file has the ID "0", I should store the both of num in each line into an data structure array. For example, we can see that num "1" has ID "0" in second file, so I need to store:

1   1217907
1   1217908
1   1517737
1   2

from my first file into the data structure array. The num "2" has ID"1" but num "3" has ID "0" and num "4" has ID "1", so need to store : 2 3 but not store 2 4 from my first file. That's why I need use array to store the two files. If I use two arrays to store them, I can check whether this num's ID is "0" fast in the array belongs to second file because using array is fast to locate a specific data, the index can be the value of the num directly.

Answer 1

I think, you need to make use of snprintf() to generate the string to be passed to popen() first and then you can call popen() with that string.

Pseudo-code

char buf[32] = {0};
snprintf(buf, 32, "wc -l < %s", myfile);
fp = popen(buf, "r");

EDIT

To make it work for any length of myfile

int len = strlen(myfile) + strlen("wc -l < ") + 1;
char *buf = malloc(len);
snprintf(buf, len, "wc -l < %s", myfile);
fp = popen(buf, "r");

...

free(buf);

Note: As mentioned by Ed Heal in the comment, the 32 here is used here for just demo purpose. You should choose your temporary array length based on the length of the string held by myfile, plus the mandatory characters, plus null terminator, obviously.

Answer 2

Forget popen - do it yourself

i.e.

FILE *f = fopen(argv[1], "r");
int lines = 0;
int ch;
while ((ch = fgetc(f)) != EOF) {
   if (c == '\n') lines++;
}

EDIT - As the poster wants to load the whole file into memory

Add the checking for errors

FILE *f = fopen(argv[1], "r");
struct stat size;
fstat(fileno(f), &size);

char buf = malloc(size.st_size)
fread(buf, size.st_size, 1, f);
fclose(f);

Answer 3

If you're not going to do this yourself (without a shell), which you should, at least pass the filename in such a way that the shell will only ever interpret it as data rather than code to avoid potential for security incidents.

setenv("filename", "myfile");            /* put filename in the environment */
fp = popen("wc -l <\"$filename\"", "r"); /* check it from your shell script */

Answer 4

All of the code below is untested. If I find time to test, I'll remove this caveat.

You can create your own wrapper to popen() to allow you to form an arbitrary command.

FILE * my_popen (const char *mode, const char *fmt, ...) {
    va_list ap;
    int result = 511;

    for (;;) {
        char buf[result+1];

        va_start(ap, fmt);
        result = vsnprintf(buf, sizeof(buf), fmt, ap);
        va_end(ap);

        if (result < 0) return NULL;
        if (result < sizeof(buf)) return popen(buf, mode);
    }

    /* NOT REACHED */
    return NULL;
}

Then, you can call it like this:

const char *filename = get_filename_from_input();
FILE *fp = my_popen("r", "%s < %s", "wc -l", filename);
if (fp) {
  /* ... */
  pclose(fp); /* make sure to call pclose() when you are done */
}

Here, we assume that get_filename_from_input() transforms the filename input string into something safe for the shell to consume.

---

It is rather complex (and error prone) to reliably fix up a filename into something the shell will treat safely. It is more safe to open the file yourself. However, after doing so, you can feed the file to a command, and then read out the resulting output. The problem is, you cannot use popen() to accomplish this, as standard popen() only supports unidirectional communication.^†

†_{Some variations of popen() exist that support bidirectional communication.}

FILE * my_cmd_open (const char *cmd) {
    int s[2], p, status, e;
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, s) < 0) return NULL;
    switch (p = fork()) {
    case -1: e = errno; close(s[0]); close(s[1]); errno = e; return NULL;
    case 0: close(s[0]); dup2(s[1], 0); dup2(s[1], 1); dup2(s[1], 2);
            switch (fork()) {
            case -1: exit(EXIT_FAILURE);
            case 0: execl("/bin/sh", "-sh", "-c", cmd, (void *)NULL);
                    exit(EXIT_FAILURE);
            default: exit(0);
            }
    default: for (;;) {
                 if (waitpid(p, &status, 0) < 0 && errno == EINTR) continue;
                 if (WIFEXITED(status) && WEXITSTATUS(status) == 0) break;
                 close(s[0]); close(s[1]); errno = EPIPE;
                 return NULL;
             }
    }
    close(s[1]);
    return fdopen(s[0], "r+");
}

---

To efficiently read an entire file into memory, you can use mmap().

void * mmap_filename (const char *filename, size_t *sz) {
    int fd = open(filename, O_RDONLY);
    if (fd < 0) return NULL;
    struct stat st;
    if (fstat(fd, &st) < 0) {
        close(fd);
        return NULL;
    }
    *sz = st.st_size;
    void *data = mmap(NULL, *sz, PROT_READ, MAP_PRIVATE, fd, 0);
    close(fd);
    return data != MAP_FAILED ? data : NULL;
}

Then, you can call it like this:

size_t sz;
void *data = mmap_filename(filename, &sz);
if (data) {
    /* ... */
    munmap(data, sz);
}

The example code above maps the entire file at once. However, the mmap() API allows you to map portions of the file from a particular offset into the file.