Combining string arguments from input file and command string with GNU parallel - gnu-parallel

Question

I am trying to pass both a command string and separate arguments from an input file to GNU parallel. My script looks like this:
parallel="parallel --delay 0.2 -j 100 --joblog remaining_runs_$1.log --resume "
$srun $parallel {python3 scaling.py {1} {2} {3}} < missing_runs_$1.txt
The python script takes 3 separate integers as arguments, each listed in missing_runs_$1.txt like so:
1 1 153
1 1 154
1 1 155
1 1 156
1 1 157
1 1 158
...
I have tried using --colsep but it results to only the file arguments being passed to parallel missing the python3 scaling.py part. Without --colsep each file line is interpreted as a string which is not what I want either (e.g., python3 scaling.py '1 1 153'). Any ideas?

Answer 1

With base in your input sample, I created a reproducible example to test this issue:
A simple python script:
#!/usr/bin/python
import sys
for i in range(1, len(sys.argv)):
print(f'The argument number {i} is {sys.argv[i]}.')
And a simplified command line:
parallel --dry-run -j 100 --colsep ' ' ./python.py {1} {2} {3} :::: < missing_runs_1.txt
./python.py 1 1 153
./python.py 1 1 154
./python.py 1 1 155
./python.py 1 1 156
./python.py 1 1 157
./python.py 1 1 158
without --dry-run:
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 153.
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 154.
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 155.
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 156.
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 157.
The argument number 1 is 1.
The argument number 2 is 1.
The argument number 3 is 158.
Using all arguments from your parallel command, in the file remaining_runs_1.log, I got:
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
1 : 1630591288.009 0.021 0 86 0 0 ./python.py 1 1 153
2 : 1630591288.220 0.040 0 86 0 0 ./python.py 1 1 154
3 : 1630591288.422 0.035 0 86 0 0 ./python.py 1 1 155
4 : 1630591288.649 0.041 0 86 0 0 ./python.py 1 1 156
5 : 1630591288.859 0.042 0 86 0 0 ./python.py 1 1 157
6 : 1630591289.081 0.040 0 86 0 0 ./python.py 1 1 158
I think this can solve the problem or at least give new ideas for the definitive solution.

Answer 2

If
parallel --delay 0.2 -j 100 --joblog curtailment_scaling_remaining_$1.log --resume python3 scaling.py {1} {2} {3} :::: < missing_runs_$1.txt
gives you:
python3 curtailment_scaling.py '1 1 163'
and you want:
python3 curtailment_scaling.py 1 1 163
you can do (version > 20190722):
parallel --delay 0.2 -j 100 --joblog curtailment_scaling_remaining_$1.log --resume python3 scaling.py {=uq=} < missing_runs_$1.txt
(uq runs uq(); which causes the replacement string not to be quoted.)
or:
parallel --delay 0.2 -j 100 --joblog eval curtailment_scaling_remaining_$1.log --resume python3 scaling.py {} < missing_runs_$1.txt