I am processing a text file with multiple parallel processes spawned by xargs. I also need to capture the stdout from each process into a separate log file. Below is an example where the output from each process is interleaved into a single file -- not what I want.
Ideally, each logfile should be numbered by the file line number, that is, logfile-1, logfile-2, etc.
cat inputfile.txt | xargs -n 1 -P 8 ./myScript.sh | tee logfile
It would be nice to avoid an external wrapper script if possible, but if there is a way to wrap myScript with a here document, that would work.
nl inputfile.txt | xargs -n 2 -P 8 sh -c './myScript.sh "$1" > logfile-$0'
This assumes each argument in
inputfile.txt is on its own line and contains no spaces. The
nl command numbers each line, which pairs each argument with a unique number. The
xargs commands takes two arguments at time, the first the line number, the second the corresponding line from
inputfile.txt, and passes them to
sh command uses the arguments to generate the output file name and the argument to