Parallel run with jenkins - jenkins

Question

I have several (few hundreds) of files to run test on (each test takes few minutes).
Running sequentially is not acceptable and neither all together. So I am looking for something like a producer-consumer.
I tried pipeline jobs and parallel command the following way:
def files = findFiles glob: 'test_files/*'
def branches = [:]
files.each{
def test_command = "./test ${it}"
branches["${it}"] = { sh "${test_command} ${it}"}
}
stage name:'run', concurrency:2
parallel branches
Problem:
All the tasks are launch at the same time (OOM and all the fun)

Answer 1

Doesn't have the same introspection as the Jenkins parallel step, but since it seems not to support a fixed pool you can use xargs to achieve the same result:
def files = findFiles glob: 'test_files/*'
def branches = [:]
// there's probably a more efficient way to generate the list of commands
files.each{
sh "echo './test ${it}' >> tests.txt"
}
sh 'cat tests.txt | xargs -L 1 -I {} -P 2 bash -c "{}"'
The -P argument is the one that specifies a fixed number of 2 (or N) processes should always be running. Other tools like GNU Parallel offer even more tuning on how many processes should be used.
You can also try to use the lock step from the Lockable Resources plugin, the node step targeting a fixed number of executors. However this seems too much overhead to me unless your single tests are already taking tens of second each.