2

I have a file with a list of paths. e.g.:

/run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/ /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/

I have a program (let's call it 'prog') which takes 3 parameters, 2 input filenames and 1 output filename. Normal execution:

prog ~/A.txt ~/B.txt ~/out.txt

I want to execute in parallel my program with parameters constructed based on each line of the file, e.g.:

prog /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/A.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/B.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/out.txt
prog /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/A.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/B.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/out.txt

I have been trying to use xargs and GNU parallel, but I can't construct a proper command line for those tools. Here is my attempt:

cat ListPaths.txt | xargs -I '{}' -n 1 -P 8 ./SkullSegmenter/SkullSegmenter '{}'A.txt '{}'B.txt '{}'out.txt

But this does not construct the arguments properly. Can someone help me with this?

Dženan
  • 3,329
  • 3
  • 31
  • 44

1 Answers1

2

This should do it:

cat ListPaths.txt | parallel ./SkullSegmenter/SkullSegmenter {}A.txt {}B.txt {}out.txt

Test with --dry-run to see if these are the commands you want run:

cat ListPaths.txt | parallel --dry-run ./SkullSegmenter/SkullSegmenter {}A.txt {}B.txt {}out.txt
Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • I run: `cat ListPaths.txt | parallel -j 8 ./SkullSegmenter/SkullSegmenter {}SubjectBoneWithoutSuturesIndividualV2.nrrd {}RegisteredSegmentedTemplateV2.nrrd {}SegmentedSubjectV2.nrrd` I get: `SubjectBoneWithoutSuturesIndividualV2.nrrd7/gvfs/smb-share:server=xeonator,share=c/Results/Normals/18_mo/CNMC_N_118/ Exception Line: 143 SubjectBoneWithoutSuturesIndividualV2.nrrdor reading file /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/18_mo/CNMC_N_118/ The file doesn't exist. SubjectBoneWithoutSuturesIndividualV2.nrrderver=xeonator,share=c/Results/Normals/18_mo/CNMC_N_118/` – Dženan Aug 27 '15 at 14:41
  • It looks like only the path from the file is passed as an argument, and it is not merged with file names A, B and out. – Dženan Aug 27 '15 at 14:45
  • Since I do not use SkullSegmenter, can you re-test with the command `echo` and maybe use some short input, so it is not so hard to decipher? Also: what did the `--dry-run` give? – Ole Tange Aug 27 '15 at 18:36
  • I tried it again, this time it is working - the paths are passed correctly. Meanwhile I got sudo priviledge so I could mount that share in a more sane location (`~/y`). Passing paths starting with `~/y/` did not work (file not found), but writing full paths (`/home/dzenan/y/`) did. Does running this stuff through SSH make a difference? Anyway, thanks for help @Ole Tange – Dženan Aug 27 '15 at 20:10
  • GNU Parallel quotes the input, so if the input is `~/y` then GNU Parallel treats this as `~/y` and not `your_homedir/y`. You may be able to force the shell to dequote it by prepending the command to run with `eval`. – Ole Tange Aug 27 '15 at 21:07