1

Let's say I had 10MB string to pass to a CLI command.

Could I directly pass it in the CLI? eg. my_cmd "my_string" Or maybe my_cmd << file_with_data?

Or would I have to write it to disk first and then the command internally read the file?

Could I pass the command a memory location of the string?

What other options exist?

====

EDIT: To be clear my question is about passing a huge string through a CLI. Are there size limits? Other issues?

EDIT 2: I would be passing this data from a program to a child process, written in different languages.

Anton
  • 2,282
  • 26
  • 43
  • What is your question? Of course it's possible to feed a cli command with a string? PLease show us an example of the real issue. – 0stone0 Aug 25 '20 at 12:29
  • Made an edit. @0stone0 – Anton Aug 25 '20 at 12:32
  • I've tried to do something similar before and I remember running into issues. I think it depends on either the OS or the shell, but there is generally a buffer with a limit somewhere. See [here](https://stackoverflow.com/questions/19354870/bash-command-line-and-input-limit). I would definitely just pass the data through a file. – 0x5453 Aug 25 '20 at 12:43
  • On what system are you? AFAIK Posix says at least 4096. – 12431234123412341234123 Aug 25 '20 at 12:44
  • Related: https://stackoverflow.com/questions/14419464/c-argv-what-is-the-maximum-size-of-data and https://stackoverflow.com/questions/7498892/about-command-line-arguments-of-main-function – 12431234123412341234123 Aug 25 '20 at 12:46
  • 1
    From where do you get the data? If they are in a file, why not read the file. If you give them from one program to a child, you could try using shared memory, pipes or sockets. – 12431234123412341234123 Aug 25 '20 at 12:48
  • From a program to a child. – Anton Aug 25 '20 at 12:49
  • stdin and stdout would be the simplest, a socket would be more advanced but has some advantages. Shared memory is a bit harder when you use different languages. – 12431234123412341234123 Aug 25 '20 at 13:00

2 Answers2

1

10MB shouldn't be an issue, you could verify this like so;

Use the following command to generate 10MB random data: base64 /dev/urandom | head -c 10000000, now use <<< to feed it to your command;

wc -l <<< $(base64 /dev/urandom | head -c 10000000) # 0.42s

grep -i 'z' <<< $(base64 /dev/urandom | head -c 10000000) # 0.61s

I'm using the unix time to check the duration

If you wish to (temporary) write the data to a file, use: base64 /dev/urandom | head -c 10000000 > file_10_mb.txt

wc -l < file_10_mb.txt # 0.01s

grep 'z' < file_10_mb.txt | wc -l # 0.07

If your my_cmd doesn't require a lot of resources, 10MB should work just fine

0stone0
  • 34,288
  • 4
  • 39
  • 64
  • Can I do the same without first creating a file? I would like to skip the step of writing to disk for performance reasons. – Anton Aug 25 '20 at 12:49
  • I think you missunderstood the question. The question was if an argument of the size of 10 MB can be passed. Basically `command "$(cat <10Mb file>)"` without a problem. – 12431234123412341234123 Aug 25 '20 at 12:50
0

To get an answer to the limit see the related question and answers here: About command line arguments of main function It seems like 10 MB is a bit large and much larger than the minimum for Posix which is 4096. On my AMD64 Linux the limit is about 2 MiB.

For better options: There are files, pipes (including stdin and stdout), sockets or shared memory.

12431234123412341234123
  • 2,732
  • 15
  • 22