I am trying to utilize a new feature in 19.3 per this question: Erlang: does the application behavior trap SIGTERM?
My understanding is that sending SIGTERM to BEAM now triggers a graceful shutdown in Erlang 19.3+
I start my application in Docker using the ENTRYPOINT ./_build/default/rel/myapp/bin/myapp where ./_build/default/rel/myapp/bin/myapp is generated from rebar3 release
When I do this in Docker, myapp gets PID1 and BEAM seems to gets another PID.
Is there a different set of commands I can run such that BEAM gets PID1 and myapp gets loaded from there? Something like
./start_beam; ./start_my_app_via_beam?
I need this because docker stop sends SIGTERM to the PID1. I need that to be BEAM. Using the above entrypoint, here is what happens in the container":
top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 4340 644 556 S 0.0 0.0 0:00.01 myapp
14 root 20 0 3751188 50812 6660 S 0.0 0.6 0:00.48 beam.smp
18 root 20 0 11492 116 0 S 0.0 0.0 0:00.00 epmd
31 root 20 0 4220 680 604 S 0.0 0.0 0:00.10 erl_child_setup
53 root 20 0 11456 944 840 S 0.0 0.0 0:00.00 inet_gethost
54 root 20 0 17764 1660 1504 S 0.0 0.0 0:00.00 inet_gethost
55 root 20 0 20252 3208 2720 S 0.0 0.0 0:00.02 bash
61 root 20 0 21956 2468 2052 R 0.0 0.0 0:00.00 top
Currently, to get around this, I have this horrendous beast:
#!/usr/bin/env bash
echo "if testing locally send SIGTERM to $$"
term_handler() {
echo "Stopping the Erlang VM gracefully"
#/usr/local/Cellar/erlang/19.1/lib/erlang/lib/erl_interface-
3.9.1/bin/erl_call -c myapp -s -a 'init stop' -n 'myapp#localhost'
/usr/local/lib/erlang/lib/erl_interface-3.9.2/bin/erl_call -c myapp -s -a 'init stop' -n 'myapp#localhost'
echo "Erlang VM Stopped"
}
trap term_handler SIGQUIT SIGINT SIGTERM
./_build/default/rel/myapp/bin/myapp &
PID=$!
echo "Erlang VM Started"
#wait $PID
while kill -0 $PID ; do wait $PID ; EXIT_STATUS=$? ; done
echo "Exiting Wrapper."
exit $EXIT_STATUS
```
And then I do `ENTRYPOINT : ["./thisscript"]`
This beast becomes PID 1, and it finds the correct thing to kill after that.
I'm trying to get rid of this script.
I have this Dockerfile (where I am using miniconda just because I would like to schedule some python scripts, but it's a debian:jessie docker image):
FROM continuumio/miniconda:4.2.12
RUN mkdir -p /workspace
WORKDIR /workspace
ADD volume .
RUN apt-get update
RUN apt-get install -y cron
ENTRYPOINT ["/bin/sh", "/workspace/conf/entrypoint.sh"]
The script entrypoint.sh that keeps the container alive is this one:
#!/usr/bin/env bash
echo ">>> Configuring cron"
service cron start
touch /var/log/cron.log
mv /workspace/conf/root /var/spool/cron/crontabs/root
chmod +x /var/spool/cron/crontabs/root
crontab /var/spool/cron/crontabs/root
echo ">>> Done!"
tail -f /var/log/cron.log
From the docker documentation about supervisor (https://docs.docker.com/engine/admin/using_supervisord/) it looks like that could be an option as well as the bash script option (as in my example), that's why I decided to go for the bash script and to ignore supervisor.
And the content of the cron details /workspace/conf/root is this:
* * * * * root echo "Hello world: $(date +%H:%M:%S)" >> /var/log/cron.log 2>&1
(with at the bottom as an empty line \n)
I can not find a way to see that Hello world: $(date +%H:%M:%S) each minute appended to /var/log/cron.log, but to me all the cron/crontab settings are correct.
When I check the logs of the container I can see:
>>> Configuring cron
[ ok ] Starting periodic command scheduler: cron.
>>> Done!
Also, when logging into the running container I can see the cron daemon running:
root#2330ced4daa9:/workspace# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4336 1580 ? Ss+ 13:06 0:00 /bin/sh /workspace/conf/entrypoint.sh
root 14 0.0 0.0 27592 2096 ? Ss 13:06 0:00 /usr/sbin/cron
root 36 0.0 0.0 5956 740 ? S+ 13:06 0:00 tail -f /var/log/cron.log
root 108 0.5 0.1 21948 3692 ? Ss 13:14 0:00 bash
root 114 0.0 0.1 19188 2416 ? R+ 13:14 0:00 ps aux
What am I doing wrong?
Are you sure the Cronjob has execution rights?
chmod 0644 /var/spool/cron/crontabs/root
I have a Docker image based on Ubuntu that runs a supervisor script as the CMD at the end of the Dockerfile. This successfully runs uwsgi and nginx in the container on start up. However, the following appended at the end of the supervisor-app.conf does not work:
[program:Xvfb]
command=/usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
When I open a shell into a running docker instance there is no X instance running:
root#9221694363ea:/# ps aux | grep X
root 39 0.0 0.0 8868 784 ? S+ 15:32 0:00 grep --color=auto X
However, running exactly the same command as in the supervisor-app.conf works
root#9221694363ea:/# /usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
[1] 40
root#9221694363ea:/# ps aux | grep X
root 40 1.2 0.1 170128 21604 ? Sl 15:33 0:00 /usr/bin/Xvfb :1 -screen 0 1024x768x16
root 48 0.0 0.0 8868 792 ? S+ 15:33 0:00 grep --color=auto X
so what's wrong with the line in the supervisor-app.conf?
Supervisor does not handle bash specific operators such as the-run-in-the -background '&' or redirections like '>' as per my original failing config line.
I solved it by using bash -c thus:
[program:Xvfb]
command=bash -c "/usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log"
Now when I get into the docker bash shell the Xvfb window is created waiting for me to use it elsewhere in the code.
I would like to search from the output of ps command by using grep. The criteria is: it contains the specific username and command.
For instance, with the following ps result:
suh1 48980 0.0 0.0 144396 20372 ? Ss Mar16 0:20 SCREEN -DR l2
tprossi 52257 0.0 0.0 127344 3264 ? Ss Mar13 0:26 SCREEN
mannerh1 54331 0.0 0.0 125368 1288 ? Ss 14:32 0:00 SCREEN
thakorv1 54791 0.0 0.0 126096 2116 ? Ss Mar16 0:00 SCREEN
valiman2 58699 0.0 0.0 125364 1244 ? Ss Mar17 0:00 SCREEN
rxue 60121 4.0 0.0 108344 1120 pts/176 R+ 14:39 0:00 ps aux
rxue 60122 0.0 0.0 103252 920 pts/176 S+ 14:39 0:00 grep -e rxue -e SCREEN
I would like to get the process list with user "tprossi" and with command SCREEN.
One solution is:
ps aux | grep tprossi | grep -i screen
But here the grep is called twice, which is not so elegant. Is there any shortcut?
I tried also:
ps aux | grep -i "tprossi*screen"
but it doesn't work :<
Anyone can give some suggestion? Thanks in advance!
You miss a .
ps aux | grep -i "tprossi.*screen"
Without it, you just say repeat i zero or more times.
The . represent any character.
You can also use awk
ps aux | awk '/tprossi/ && /screen/' # in any order
or
ps aux | awk '/tprossi.*screen/' # in this order
I am trying to write a Fortran program which will eat up a lot of memory (for the reasoning behind this, please see the note at the end of this question). I am doing this by allocating a 3 dimensional array of size (n,n,n) and then deallocating it - continually increasing n until I run out of memory (this should happen when ~16 GB of memory is used). Unfortunately, it seems as if my program is running out of memory long before I see the system resources get up to 16 GB.
Here is my sample code:
1 program fill_mem
2 implicit none
3 integer, parameter :: ikind = selected_int_kind(8)
4 integer, parameter :: rkind = 8
5
6 integer(kind = ikind) :: nfiles = 100
7 integer(kind = ikind) :: n = 1200
8 integer(kind = ikind) :: i, nn
9
10 real(kind = rkind), allocatable :: real_arr(:,:,:)
11
12 character(500) :: sysline
13
14
15 call system('echo ''***no_allocation***'' > outfile')
16 call system('ps aux | grep fill_mem.exe >> outfile')
17 !call system('smem | grep fill_mem.exe >> sm.out')
18 allocate(real_arr(n, n, n))
19
20 nn = 100000
21 do i = 1,nn
22 deallocate(real_arr)
23 n = n + 10
24 print*, 'n = ', n
25 allocate(real_arr(n, n, n))
26 call system('echo ''*************'' >> outfile')
27 write(sysline, *) 'allocation', i, '... n = ', n
28
29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%'
30
31 call system(trim(adjustl('echo '//sysline//'>> outfile')))
32 call system('ps aux | grep fill_mem.exe >> outfile')
33 enddo
34
35 end program fill_mem
and here is the sample output:
1 ***no_allocation***
2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe
3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
5 *************
6 allocation 1 ... n = 1210
7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe
8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
10 *************
11 allocation 2 ... n = 1220
12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe
13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
15 *************
16 allocation 3 ... n = 1230
17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
20 *************
21 allocation 4 ... n = 1240
22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
25 *************
26 allocation 5 ... n = 1250
27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe
30 *************
31 allocation 6 ... n = 1260
32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
35 *************
36 allocation 7 ... n = 1270
37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
Now, I see that the VSZ portion gets up to ~15 GB so I am assuming when I try to address more, it fails with
Operating system error: Cannot allocate memory
Allocation would exceed memory limit
because there is not that much RAM. Why is it that RSS is so far below that, though? When I actually look on my system resources I see about 140 MB being used up (I am running this in a Linux VM and monitoring the system resources through Windows - I have given the GM 16 GB of RAM to use though, so I should see the VM memory increasing until it reaches the 16 GB mark - for what it's worth, the VM has VT-x/Nested Paging/PAE/NX so it should use the physical architecture just like the native OS).
Can anyone explain why I do not see my program actually using up the full 16 GB of RAM and how I can write my code to keep these arrays I am creating in RAM - fully utilizing my available hardware?
NOTE: The reason I am trying to write a sample program which reads a lot of memory is that I am working with data which takes up around 14 GB of space in ascii text. I will need to be working with data A LOT throughout the course of this program, so I want to read it all in at once and then reference it from RAM throughout the duration of the program. To make sure I am doing this correctly, I am trying to write a simple program which will store a very large array (~15 GB) in memory all at once.
(Caveat: The Fortran standard doesn't say how such thing ought to be implemented etc., the description below refers to how Fortran compilers are typically implemented on current operating systems.)
When you execute an ALLOCATE statement (or equivalently, calling malloc() in C, FWIW), you're not actually reserving physical memory, but only mapping address space for your process. That's why the VSZ goes up, but not the RSS. Actually reserving physical memory for your process happens only when you first access the memory (typically at page size granularity, that is, 4 KB on most current hw). So only once you start putting some data into your array does the RSS begin to climb. E.g. a statement like
real_arr = 42.
ought to bump up your RSS to the vicinity of the VSZ.
You probably need to increase the memory allocated to the stack. For example, see http://software.intel.com/en-us/articles/intel-fortran-compiler-increased-stack-usage-of-80-or-higher-compilers-causes-segmentation-fault