Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
58
votes
4 answers

GNU parallel not working at all

I have been trying to use GNU parallel for some time, but I have never been able to get it to function at all! For example, running (in a non-empty directory!): ls | parallel echo # Outputs single new line ls | parallel echo echo echo #…
WaelJ
  • 2,942
  • 4
  • 22
  • 28
49
votes
4 answers

Splitting command line args with GNU parallel

Using GNU parallel: http://www.gnu.org/software/parallel/ I have a program that takes two arguments, e.g. $ ./prog file1 file2 $ ./prog file2 file3 ... $ ./prog file23456 file23457 I'm using a script that generates the file name pairs, however this…
drhodes
  • 1,009
  • 1
  • 10
  • 17
33
votes
2 answers

GNU Parallel and Bash functions: How to run the simple example from the manual

I'm trying to learn GNU Parallel because I have a case where I think I could easily parallelize a bash function. So in trying to learn, I went to the GNU Parallel manual where there is an example...but I can't even get it working! To wit: (232) $…
Fortran
  • 475
  • 1
  • 5
  • 8
19
votes
2 answers

parallel call multiple bash functions

I have read the example at http://www.gnu.org/software/parallel/man.html#example__calling_bash_functions however, is it possible to use gnu parallel to call 2 functions which do not have any variables you pass to them ? example a() { echo…
p4guru
  • 1,400
  • 2
  • 19
  • 25
18
votes
7 answers

bash add/append new columns from other files

I have a name.txt file of one column, e.g. A B C D E F Then I have many files, e.g. x.txt, y.txt and z.txt x.txt has A 1 C 3 D 2 y.txt has A 1 B 4 E 3 z.txt has B 2 D 2 F 1 The desirable output is (filling in 0 if there is no mapping) A 1 1 0 B…
Elfxy
  • 183
  • 6
15
votes
4 answers

"find" and "ls" with GNU parallel

I'm trying to use GNU parallel to post a lot of files to a web server. In my directory, I have some files: file1.xml file2.xml and I have a shell script that looks like this: #! /usr/bin/env bash CMD="curl -X POST -d@$1 http://server/path" eval…
Dave
  • 1,567
  • 4
  • 13
  • 23
15
votes
3 answers

Parallelizing a while loop with arrays read from a file in bash

I have a while loop in Bash handled like this: while IFS=$'\t' read -r -a line; do myprogram ${line[0]} ${line[1]} ${line[0]}_vs_${line[1]}.result; done < fileinput It reads from a file with this structure, for reference: foo bar baz …
Einar
  • 4,727
  • 7
  • 49
  • 64
15
votes
2 answers

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU Parallel and psql copy. The problem is that it takes…
Topo
  • 4,783
  • 9
  • 48
  • 70
14
votes
4 answers

Uploading files to s3 using s3cmd in parallel

I've got a whole heap of files on a server, and I want to upload these onto S3. The files are stored with a .data extension, but really they're just a bunch of jpegs,pngs,zips or pdfs. I've already written a short script which finds the mime type…
Alan Hollis
  • 1,292
  • 3
  • 14
  • 29
13
votes
2 answers

Feed GNU parallel with an array?

How to feed a command in GNU parallel with an array? For example, I have this array: x=(0.1 0.2 0.5) and now I want to feed it to some command in parallel parallel echo ::: $x This does not work. It is feeding all the arguments to a single call,…
a06e
  • 18,594
  • 33
  • 93
  • 169
12
votes
2 answers

How do I execute multiple commands in parallel on an array of parameters with bash, and fail if at least one of them failed

I have a bash script with a function that needs to run in parallel with different arguments. I need to know if at least one of the executions failed (returned non-zero) - doesn't matter how many failed. The command accepts an array of parameters for…
Avia Eyal
  • 123
  • 7
12
votes
5 answers

Check if a remote file exists in bash

I am downloading files with this script: parallel --progress -j16 -a ./temp/img-url.txt 'wget -nc -q -P ./images/ {}; wget -nc -q -P ./images/ {.}_{001..005}.jpg' Would it be possible to not download files, just check them on the remote side and if…
Adrian
  • 2,576
  • 9
  • 49
  • 97
12
votes
3 answers

gnu parallel: output each job to a different file

I am trying to process so text files with awk using the parallel command as a shell script, but haven't been able to get it to output each job to a different file If i try: seq 10 | parallel awk \''{ if ( $5 > 0.4 ) print $2}'\' file{}.txt >…
Scott Ritchie
  • 10,293
  • 3
  • 28
  • 64
11
votes
1 answer

Gnu Parallel : nested parallelism

Is it possible to call gnu parallel from within multiple runs of a script that are in-turn spawned by gnu parallel? I have a python script that runs for 100s of sequential iterations, and somewhere within each iteration, 4 values are being…
Neha Karanjkar
  • 3,390
  • 2
  • 29
  • 48
11
votes
4 answers

export function from zsh to bash for use in gnu parallel

How do I export a function from zsh, so that I can use it in gnu parallel? example: function my_func(){ echo $1;} export -f my_func parallel "my_func {}" ::: 1 2 in bash will output 1 2 whereas in zsh it will output error messages /bin/bash:…
Bilal Syed Hussain
  • 8,664
  • 11
  • 38
  • 44
1
2 3
…
48 49