Working off this example: http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Speeding-up-fast-jobs
When I run:
seq -w 0 9999 | parallel touch pict{}.jpg
seq -w 0 9999 | parallel -X touch pict{}.jpg
Success! However, add another 9 and BOOM:
$ seq -w 0 99999 | parallel --eta -X touch pict{}.jpg
parallel: Warning: No more processes: Decreasing number of running jobs to 3. Raising ulimit -u or /etc/security/limits.conf may help.
Computers / CPU cores / Max jobs to run
1:local / 4 / 3
parallel: Warning: No more processes: Decreasing number of running jobs to 2. Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 1. Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Error: No more processes: cannot run a single job. Something is wrong.
I would expect parallel -X to run no more jobs than I have cpu cores, and to cram as many parameters onto each job as the max command line length permits. How am I running out of processes?
My environment:
OSX Yosemite
ulimit -u == 709
GNU parallel 20141122
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin14)
Your expectation is 100% correct. What you are seeing is clearly a bug - probably due to GNU Parallel not being well tested on OSX. Please follow http://www.gnu.org/software/parallel/man.html#REPORTING-BUGS and file a bug report.
Related
Assume the following:
I have a program myprogram inside a docker container
I'm running the docker container with
docker run --privileged=true my-label/my-container
Inside the container - the program is being run with:
strace -f -e trace=desc ./myprogram
What I see is that the strace (despite having the -f on) doesn't follow all the child processes.
I see the following output from strace
[pid 10] 07:36:46.668931 write(2, "..\n"..., 454 <unfinished ...>
<stdout of ..>
<stdout other output - but I don't see the write commands - so probably from a child process>
[pid 10] 07:36:46.669684 write(2, "My final output\n", 24 <unfinished ...>
<stdout of My final output>
What I want to see is the other write commands.
Now I should see the the other write commands - because I'm using -f.
What I think is happening is that running inside docker makes the process handling and security different.
My question is: Does strace -f work differently when run inside a docker container?
Note that this application starts and stops in 2 seconds - so the tracing tool has to follow the application lifecycle - like strace does. Connecting to a server background process won't work.
It turns out strace truncates string output - you have to explicitly tell it that you want more than the first n (10?) string chars. You do this with -s 800.
strace -s 800 -ff ./myprogram
You can also get all the write commands by asking strace explicitly with -e write.
strace -s 800 -ff -e write ./myprogram
I have been working with Docker for a while now, I have installed docker and launched a container using
docker run -it --cpuset-cpus=0 ubuntu
When I log into the docker console and run
grep processor /proc/cpuinfo | wc -l
It shows 3 which are the number of cores I have on my host machine.
Any idea on how to restrict the resources to the container and how to verify the restrictions??
The issue has been already raised in #20770. The file /sys/fs/cgroup/cpuset/cpuset.cpus reflects the correct output.
The cpuset-cpus is taking effect however is not being reflected in /proc/cpuinfo
docker inspect <container_name>
will give the details of the container launched u have to check for "CpusetCpus" in there and then u will find the details.
Containers aren't complete virtual machines. Some kernel resources will still appear as they do on the host.
In this case, --cpuset-cpus=0 modifies the resources the container cgroup has access to which is available in /sys/fs/cgroup/cpuset/cpuset.cpus. Not what the VM and container have in /proc/cpuinfo.
One way to verify is to run the stress-ng tool in a container:
Using 1 cpu will be pinned at 1 core (1 / 3 cores in use, 100% or 33% depending on what tool you use):
docker run --cpuset-cpus=0 deployable/stress -c 3
This will use 2 cores (2 / 3 cores, 200%/66%):
docker run --cpuset-cpus=0,2 deployable/stress -c 3
This will use 3 ( 3 / 3 cores, 300%/100%):
docker run deployable/stress -c 3
Memory limits are another area that don't appear in kernel stats
$ docker run -m 64M busybox free -m
total used free shared buffers cached
Mem: 3443 2500 943 173 261 1858
-/+ buffers/cache: 379 3063
Swap: 1023 0 1023
yamaneks answer includes the github issue.
it should be in double quotes --cpuset-cpus="", --cpuset-cpus="0" means it make use of cpu0.
Consider MyProcess, a purely deterministic process. If I run this process several times measuring the max memory usage with usz time I get
time ./MyProcess
max memory: 86624 KB
time ./MyProcess
max memory: 85740 KB
time ./MyProcess
max memory: 86156 KB
The measures of RAM usage are quite similar of course but they differ a little bit. What create these differences?
Is it because the max memory is calculated as the maximum observed RAM usage over several point measurement through time. The max memory is actually exactly the same.
Is it because even in deterministic processes, there are small differences that can be caused for example by the RAM available when allocation of memory occurs
other reasons...
FYI, I use MACOSX 10.11.3
This is most likely due to Address Space Layout Randomization, aka ASLR.
Modern OS shift and shuffle a process's memory layout on each new execution so that attackers will not know where the interesting addresses are.
Normally programmers don't notice, except through small changes in used memory or that all pointers have different values (causing e.g. different iteration order through hash tables keyed on pointers).
Here's an example program for Linux, which also does ASLR:
#include <stdio.h>
int main() {
char c;
printf("Address: %lx\n", &c);
return 0;
}
If we run it multiple times, we see different addresses and memory usage:
$ gcc foo.c -o foo
$ command time -f 'Memory: %M' ./foo
Address: 7ffc1995f2df
Memory: 1288
$ command time -f 'Memory: %M' ./foo
Address: 7ffcdfd2427f
Memory: 1324
$ command time -f 'Memory: %M' ./foo
Address: 7ffe3022a23f
Memory: 1368
If we disable ASLR (this is Linux specific and does not work on OSX) with:
sudo tee /proc/sys/kernel/randomize_va_space <<< 0
Addresses and used memory is always the same:
$ command time -f 'Memory: %M' ./foo
Address: 7fffffffe4bf
Memory: 1272
$ command time -f 'Memory: %M' ./foo
Address: 7fffffffe4bf
Memory: 1272
$ command time -f 'Memory: %M' ./foo
Address: 7fffffffe4bf
Memory: 1272
I would like to set up a 6 shards solr collection on 3 windows machines.
Tried the bin\solr -e cloud and set up 2 machines 6 shards and 1 replica. When stopping and starting 2 cores on one machine (each using another hard disk) I get 6 shards; 3 for each core.
When I start another core on another machine nothing happens, the 3rd one doesn't do anything.
When I start another core on the same machine using the same config in another directory nothing happens, the core starts but has no collections and the 2 cores first started still have 3 shards each.
For example: I start the 3rd one with:
bin\solr start -c -p 7576 -z localhost:9983 -s server/solr/collection/node3/solr
Or start on another machine:
bin\solr start -c -p 7576 -z zookeeper:9983 -s server/solr/collection/node3/solr
Is there some documentation out there that doesn't use the "convenient" bin\solr that I'm trying to reverse engineer the entire day to figure out how to set up zookeeper/solr to add the nth solr core as a shard until 6 shards are reached?
I think I found the answer: bin\solr -e cloud starts up the cores and assignes data to them.
After running the standard bin\solr -e cloud with 2 cores, a collection with 6 shards and 1 replica I stop all bin\solr stop -all
Then copy solr-5.2.1\example\cloud\node1 as solr-5.2.1\example\cloud\node3 delete the files in solr-5.2.1\example\cloud\node3\logs and let solr-5.2.1\example\cloud\node3 have gettingstarted_shard6_replica1 (leave that file in solr-5.2.1\example\cloud\node3\solr and remove it from solr-5.2.1\example\cloud\node1\solr).
Start up 3 cores:
bin\solr start -c -p 8983 -s example\cloud\node1\solr
bin\solr start -cloud -p 7574 -z localhost:9983 -s example\cloud\node2\solr
bin\solr start -cloud -p 7575 -z localhost:9983 -s example\cloud\node3\solr
And now I can see the 3rd solr instance has gettingstarted_shard6_replica1
I'm running some batches of serial programs on a (very) inhomogeneous SLURM cluster (version 2.6.6-2), using GNU 'parallel' to do the distribution. The problem that I'm having is that some of the nodes finish their tasks a lot faster than the others, and I end up with situations like, for example, a job that's allocating 4 nodes but is only using 1 during half of the simulation.
Is there any way, without administrator privileges, to free one of these unused nodes? I can mitigate the problem by running 4 jobs on individual nodes, or with files containing lists of homogeneous nodes, but it's still far from ideal.
For reference, here are the script files that I'm using (adapted from here)
job.sh
#!/bin/sh
#SBATCH --job-name=test
#SBATCH --time=96:00:00
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1024
#SBATCH --ntasks-per-node=4
#SBATCH --partition=normal
# --delay .2 prevents overloading the controlling node
# -j is the number of tasks parallel runs so we set it to $SLURM_NTASKS
# --joblog makes parallel create a log of tasks that it has already run
# --resume makes parallel use the joblog to resume from where it has left off
# the combination of --joblog and --resume allow jobs to be resubmitted if
# necessary and continue from where they left off
parallel="parallel --delay .2 -j $SLURM_NTASKS"
$parallel < command_list.sh
command_list.sh
srun --exclusive -N1 -n1 nice -19 ./a.out config0.dat
srun --exclusive -N1 -n1 nice -19 ./a.out config1.dat
srun --exclusive -N1 -n1 nice -19 ./a.out config2.dat
...
srun --exclusive -N1 -n1 nice -19 ./a.out config31.dat
You can use the scontrol command to downsize your job:
scontrol update JobId=# NumNodes=#
I am not sure however how Slurm chooses the nodes to dismiss. You might need to choose them by hand and write
scontrol update JobId=# NodeList=<names>
See Question 24 in the Slurm FAQ.