1

There are more than 6k JSON files, each containing exactly one JSON object. I want to prepare one list of objects from these JSONs.

When I am running below jq command I am getting an error.

Kedar.Javalkar@KD2806 MINGW64 /c/zz
$ jq -s '.' inventoryItem_*.json > inventory_items_result_$(date +"%Y%m%d_%H%M%S").json
bash: /usr/bin/jq: Argument list too long

I tried ulimit -s unlimited but the same error

I am using a windows 10 git bash

peak
  • 105,803
  • 17
  • 152
  • 177
Kedar Javalkar
  • 343
  • 1
  • 5
  • 22
  • 1
    "Argument list too long" is an OS-level error -- neither bash nor jq has any control over it; there's a limited amount of space allowed for combined environment variable and argument lists. Thus, you can allow longer command lines if you reduce the size of your environment variables / export fewer of your shell variables; but it'll never be unlimited. `ulimit -s` controls stack size, but it's not the stack at fault. – Charles Duffy Jun 14 '21 at 20:08
  • @CharlesDuffy - The answers provided in the linked-to "duplicate" question do not seem to apply here, at least for the most part. No doubt there is a more relevant "duplicate" but until it is identified, I think this question should probably be reopened. – peak Jun 14 '21 at 21:13
  • @peak, I'm unclear on how the duplicate fails to apply. Could you go into more details? *Argument list too long* means that there are more arguments being passed than can be passed into the space that the `execve()` syscall is populating. `xargs` exists specifically to work around that bug. Unless `jq -s . one two; jq -s . three four` behaves differently from `jq -s . one two three four` (which would mean that `jq` behaves materially differently from `rm`/`cp`/similar commands as discussed by the duplicate, and thus justifies differentiation), I have trouble seeing how it _couldn't_ apply. – Charles Duffy Jun 14 '21 at 21:27
  • @CharlesDuffy - Yes, applying jq -s successively is quite different from applying jq -s just once. It's really as simple as that. Since the answers in the identified "duplicate" do not seem to cover this case, it would make sense to reopen this question for the time being. – peak Jun 14 '21 at 22:29
  • (Duplicate list has been pruned to the single entirely-on-point item) – Charles Duffy Jun 15 '21 at 12:05

1 Answers1

2

This is a job that xargs is created to fix -- splitting lists of items into individual command lines that are within the permitted limit.

Because running jq -s a single time is different from concatenating the results of multiple smaller runs, it's appropriate to use xargs to combine cat invocations using the manner described in the linked duplicate.

printf '%s\0' inventoryItem_*.json \
  | xargs -0 cat \
  | jq -s . \
  >"inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • 1
    I agree with @peak - this works perfectly `printf '%s\0' inventoryItem_*.json \ | xargs -0 cat | jq -s . \ >"aaaa_inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"` – Kedar Javalkar Jun 14 '21 at 21:10
  • Can you explain how it doesn't work? `printf` is a shell builtin -- it isn't subject to _any_ limit on number of files. – Charles Duffy Jun 14 '21 at 21:25
  • @CharlesDuffy - The problem is not with printf. You apparently haven't thought through the behavior of `jq -s ...`. – peak Jun 14 '21 at 22:00
  • 1
    Ahh. I had `-c` in mind, and was misremembering the meaning of `-s`. – Charles Duffy Jun 14 '21 at 22:03
  • @peak, ...btw, this is a community-wiki answer for a reason. I'm not getting any rep points (wouldn't even if it were upvoted), and there's no reason for me to be the only person editing it. – Charles Duffy Jun 14 '21 at 22:05
  • @CharlesDuffy - Yes, I was guessing you had taken a wrong turn. Thanks for updating the answer. But I still think the Q should be reopened until a more appropriate "duplicate" is identified. I searched for `cat` in the linkee but could not find it in any undeleted answer. – peak Jun 14 '21 at 22:09
  • @peak, would you agree with https://stackoverflow.com/questions/65933153/argument-list-too-long-while-slurping-json-files as a duplicate? – Charles Duffy Jun 14 '21 at 22:47
  • @CharlesDuffy - Does the `--` trouble you? – peak Jun 15 '21 at 01:47
  • @peak, the answer is more correct with the `--` than without it, as without it file names starting with dashes won't be correctly handled. – Charles Duffy Jun 15 '21 at 02:10
  • @peak That's a POSIX thing (Utility Syntax Guidelines #10), almost all standard utilities have to support it, `cat` is not an exception. – oguz ismail Jun 15 '21 at 03:49