Running parallel based on variables from file in bash-CodePudding

I've been trying to run a while-loop in parallel for a work that takes days.

I've seen other answers where parallel was implemented within a while-loop, but for that case it does work in blocks, where the next job only works after all previous jobs finished.

This is the code, which reproduced the two columns of the CSV file:

while IFS="," read fq tab
do
echo $fq
echo $tab
done < /home/samples.csv

where the csv file contain two columns with no header (column 1 and 2), where the variables are stored. For example:

a1,b1
a2,b2
a3,b3
a4,b4

I've been trying to run this in parallel so when a job is finished the other starts immediately because of the long run times.

This is the code:

while IFS="," read fq tab
do
parallel -j 1 --verbose --delay 2  "echo $fq; echo $tab" 
done < /home/samples.csv

But this produces

a1 b1 a1,b1

a1 b1 a2,b2

a1 b1 a3,b3

a1 b1 a4,b4

And not

a1   b1
a2   b2
a3   b3
a4   b4

CodePudding user response：

cat <<_EOF > samples.csv
a2,b2
a3,b3
a4,b4
_EOF

cat samples.csv | parallel --colsep , echo column 1 = {1} column 2 = {2}

If samples.csv is TAB separated, ; separated, or separated with spaces:

cat samples.csv | parallel --colsep '\t' echo column 1 = {1} column 2 = {2}
cat samples.csv | parallel --colsep ';' echo column 1 = {1} column 2 = {2}
cat samples.csv | parallel --colsep '  ' echo column 1 = {1} column 2 = {2}