Erlang: rebar3 release, start beam first? - docker

I am trying to utilize a new feature in 19.3 per this question: Erlang: does the application behavior trap SIGTERM?
My understanding is that sending SIGTERM to BEAM now triggers a graceful shutdown in Erlang 19.3+
I start my application in Docker using the ENTRYPOINT ./_build/default/rel/myapp/bin/myapp where ./_build/default/rel/myapp/bin/myapp is generated from rebar3 release
When I do this in Docker, myapp gets PID1 and BEAM seems to gets another PID.
Is there a different set of commands I can run such that BEAM gets PID1 and myapp gets loaded from there? Something like
./start_beam; ./start_my_app_via_beam?
I need this because docker stop sends SIGTERM to the PID1. I need that to be BEAM. Using the above entrypoint, here is what happens in the container":
top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 4340 644 556 S 0.0 0.0 0:00.01 myapp
14 root 20 0 3751188 50812 6660 S 0.0 0.6 0:00.48 beam.smp
18 root 20 0 11492 116 0 S 0.0 0.0 0:00.00 epmd
31 root 20 0 4220 680 604 S 0.0 0.0 0:00.10 erl_child_setup
53 root 20 0 11456 944 840 S 0.0 0.0 0:00.00 inet_gethost
54 root 20 0 17764 1660 1504 S 0.0 0.0 0:00.00 inet_gethost
55 root 20 0 20252 3208 2720 S 0.0 0.0 0:00.02 bash
61 root 20 0 21956 2468 2052 R 0.0 0.0 0:00.00 top

Currently, to get around this, I have this horrendous beast:
#!/usr/bin/env bash
echo "if testing locally send SIGTERM to $$"
term_handler() {
echo "Stopping the Erlang VM gracefully"
#/usr/local/Cellar/erlang/19.1/lib/erlang/lib/erl_interface-
3.9.1/bin/erl_call -c myapp -s -a 'init stop' -n 'myapp#localhost'
/usr/local/lib/erlang/lib/erl_interface-3.9.2/bin/erl_call -c myapp -s -a 'init stop' -n 'myapp#localhost'
echo "Erlang VM Stopped"
}
trap term_handler SIGQUIT SIGINT SIGTERM
./_build/default/rel/myapp/bin/myapp &
PID=$!
echo "Erlang VM Started"
#wait $PID
while kill -0 $PID ; do wait $PID ; EXIT_STATUS=$? ; done
echo "Exiting Wrapper."
exit $EXIT_STATUS
```
And then I do `ENTRYPOINT : ["./thisscript"]`
This beast becomes PID 1, and it finds the correct thing to kill after that.
I'm trying to get rid of this script.

Related

How to control the count of processes spawned by ng ng build --prod=true to avoid bitbucket pipeline from failing with 'Build' exceeded memory limit

I am doing a memory dump on the instance where ng build is triggered. I can show the time where number of processes are spawned by ng build. Is there way to control this number.
total used free shared buff/cache available
Mem: 30G 8.5G 2.7G 215M 19G 21G
Swap: 0B 0B 0B
Fri Jan 27 15:07:19 UTC 2023
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4288 708 ? Ss 15:02 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 8 0.0 0.0 4288 96 ? S 15:02 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 9 0.0 0.0 4200 716 ? S 15:02 0:00 /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result
root 11 0.0 0.0 4288 1460 ? Ss 15:02 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperScript14257846929627798257.sh
root 35 0.0 0.0 4288 764 ? S 15:02 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/buildScript1578831321044327918.sh
root 36 0.0 0.0 18004 2936 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 37 0.0 0.0 18008 2416 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 38 0.0 0.0 18008 2416 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 40 0.0 0.0 18024 2924 ? S 15:02 0:00 bash docker-build.sh
root 60 0.0 0.1 666880 41336 ? Sl 15:02 0:00 npm
root 71 0.0 0.0 4296 808 ? S 15:02 0:00 sh -c npm i --unsafe-perm -g #angular/cli && npm i && npm run copyi18n && npm run bump_version && node --max_old_space_size=6656 node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 327 125 10.3 4375844 3362764 ? Rl 15:03 5:00 ng build --prod=true --base-href=/
root 574 0.0 0.0 4196 680 ? S 15:07 0:00 sleep 5
root 576 0.0 0.0 36644 2844 ? R 15:07 0:00 ps aux
In the above there is PID - 317 and only one ng build instance
After 10 second i see the following
Fri Jan 27 15:07:29 UTC 2023
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4288 708 ? Ss 15:02 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 8 0.0 0.0 4288 96 ? S 15:02 0:00 /bin/sh -c exit $( (/usr/bin/mkfifo /opt/atlassian/pipelines/agent/tmp/build_result && /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result) || /bin/echo 1)
root 9 0.0 0.0 4200 716 ? S 15:02 0:00 /bin/cat /opt/atlassian/pipelines/agent/tmp/build_result
root 11 0.0 0.0 4288 1460 ? Ss 15:02 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/wrapperScript14257846929627798257.sh
root 35 0.0 0.0 4288 764 ? S 15:02 0:00 /bin/sh /opt/atlassian/pipelines/agent/tmp/buildScript1578831321044327918.sh
root 36 0.0 0.0 18004 2936 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 37 0.0 0.0 18008 2416 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 38 0.0 0.0 18008 2416 ? S 15:02 0:00 /bin/bash -i /opt/atlassian/pipelines/agent/tmp/bashScript16516797543053797206.sh
root 40 0.0 0.0 18024 2924 ? S 15:02 0:00 bash docker-build.sh
root 60 0.0 0.1 666880 41336 ? Sl 15:02 0:00 npm
root 71 0.0 0.0 4296 808 ? S 15:02 0:00 sh -c npm i --unsafe-perm -g #angular/cli && npm i && npm run copyi18n && npm run bump_version && node --max_old_space_size=6656 node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 327 123 12.5 5146756 4055172 ? Sl 15:03 5:07 ng build --prod=true --base-href=/
root 578 40.1 0.2 611084 90388 ? Rl 15:07 0:02 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 585 86.2 0.3 651124 129216 ? Rl 15:07 0:04 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 596 25.6 0.3 638388 117496 ? Rl 15:07 0:01 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 604 29.0 0.2 609828 88928 ? Rl 15:07 0:01 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 611 102 0.2 617016 95864 ? Rl 15:07 0:04 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 618 32.7 0.3 620824 99604 ? Rl 15:07 0:01 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 625 66.2 0.4 657444 136912 ? Rl 15:07 0:02 /usr/local/bin/node --max_old_space_size=6656 /opt/atlassian/pipelines/agent/build/node_modules/worker-farm/lib/child/index.js /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/#angular/cli/bin/ng build --prod=true --base-href=/
root 633 0.0 0.0 4196 652 ? S 15:07 0:00 sleep 5
root 635 0.0 0.0 36644 2804 ? R 15:07 0:00 ps aux
After this point the available RAM Drops and then eventually fails
total used free shared buff/cache available
Mem: 30G 11G 762M 169M 18G 19G
Swap: 0B 0B 0B
Bitbucket Pipeline File
branches:
master:
- step:
size: 2x
script:
- bash docker-build.sh
'{dev/*,release/*,hotfix/*}':
- step:
size: 2x
script:
- while true; do date && ps aux && echo "" && sleep 5; done &
- while true; do free -h && echo "" && sleep 5; done &
- bash docker-build.sh
definitions:
services:
node:
image: node:10.15.3
memory: 7680
docker:
memory: 512
# Docker true for running docker daemon commands. By default it will be there in step
options:
docker: true
size: 2x
package.json snippet
{
"name": "de-ui",
"version": "4.7.1",
"scripts": {
"ng": "ng",
"start": "node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng serve",
"build": "ng build",
"test": "ng test",
"lint": "ng lint",
"e2e": "ng e2e",
"prod_build": "npm i --unsafe-perm -g #angular/cli && npm i && npm run copyi18n && npm run bump_version && node --max_old_space_size=6656 node_modules/#angular/cli/bin/ng build --prod=true --base-href=/",
"copyi18n": "node ./load.po.files.js ./src/assets/i18n/po/ ./src/assets/i18n/",
"createi18npo": "node ./load.po.files.js ./src/assets/i18n/ ./src/assets/i18n/po/",
"update_de": "npm update #de/de-ui-core #de/de-jsf-form #de/de-ui-app #de/de-ui-api",
"bump_version": "node ./bump_version.js",
"serve_prod": "node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng serve --prod=true"
}
Note:
Size: 2x -- 8 GB Available
docker - Set to true
Sizing:
node:
image: node:10.15.3
memory: 7680
docker:
memory: 512
node --max_old_space_size=6656 -- provided in ng build
1. How can i avoid so many process from getting triggered
2. Is there way i can re arrange the memory allocation to avoid getting --> Container 'Build' exceeded memory limit.
Have tried changing the memory sizing. But not able to get it.
Thinking if i the number of processes that is getting spawned can be controlled, then can handle memory issue.

Why do cron commands inside a docker container show up in log but don't actually run?

Dockerfile
FROM almalinux:8
# [... supervisord setup ...]
RUN dnf install -y \
crontabs
RUN sed -ri '/-session(\s+)optional(\s+)pam_systemd.so/d' /etc/pam.d/system-auth && \
sed -ri '/^[^#]/ s/systemd//g' /etc/nsswitch.conf
COPY $TEMPLATE_DIR/supervisord/crond.conf /etc/supervisord.d/crond.conf
crond.conf
[program:crond]
command=/usr/sbin/crond -nsm off
stdout_logfile_maxbytes=0
stdout_logfile=/dev/stdout
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
syslog:
2022-06-13 18:39:07,939 INFO success: crond entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
crontab -e
* * * * * touch /var/www/html/test.txt
syslog:
Jun 13 18:57:59 e60d29fd100e crontab[331]: (root) BEGIN EDIT (root)
Jun 13 18:58:15 e60d29fd100e crontab[331]: (root) REPLACE (root)
Jun 13 18:58:15 e60d29fd100e crontab[331]: (root) END EDIT (root)
Jun 13 18:59:01 e60d29fd100e CROND[334]: (root) CMD (touch /var/www/html/test.txt)
I thought the file is never touched... tried also echo, running an absolute path command... nothing. But after waiting a little (longer) and running cron with debug flags on, it seems the command does get run, but with something like a 5s to 50s delay:
load_entry()...about to parse command
2022-06-13T16:17:01.352923611Z linenum=21
2022-06-13T16:17:01.352929349Z load_entry()...returning successfully
2022-06-13T16:17:01.352934854Z ...load_user() done
2022-06-13T16:17:01.352940748Z unlinking old database:
2022-06-13T16:17:01.352960040Z check_inotify_database is done
2022-06-13T16:17:01.352966537Z user [root:0:0:...] cmd="touch /var/www/html/test.txt"
2022-06-13T16:17:01.352972511Z [10] do_command(touch /var/www/html/test.txt, (root,0,0))
2022-06-13T16:17:01.352979069Z [10] main process returning to work
2022-06-13T16:17:01.352984792Z
The huge delay seems to also pile up what I presume are queued commands to run, and the pile grows forever larger:
root 335 0.0 0.0 69708 5188 ? S 19:15 0:00 /usr/sbin/CROND -nsm off -x ext,sch,proc,pars,load,misc
root 336 94.4 0.0 69708 1440 ? Rs 19:15 8:52 /usr/sbin/CROND -nsm off -x ext,sch,proc,pars,load,misc
... multiply 10x the 2 processes above after a few minutes ...
Any clues why the huge delay and weird behavior? Disabling inotify (-i) on crond does not improve things... I'm thinking maybe a time skew issue?

cron task in docker container not being executed

I have this Dockerfile (where I am using miniconda just because I would like to schedule some python scripts, but it's a debian:jessie docker image):
FROM continuumio/miniconda:4.2.12
RUN mkdir -p /workspace
WORKDIR /workspace
ADD volume .
RUN apt-get update
RUN apt-get install -y cron
ENTRYPOINT ["/bin/sh", "/workspace/conf/entrypoint.sh"]
The script entrypoint.sh that keeps the container alive is this one:
#!/usr/bin/env bash
echo ">>> Configuring cron"
service cron start
touch /var/log/cron.log
mv /workspace/conf/root /var/spool/cron/crontabs/root
chmod +x /var/spool/cron/crontabs/root
crontab /var/spool/cron/crontabs/root
echo ">>> Done!"
tail -f /var/log/cron.log
From the docker documentation about supervisor (https://docs.docker.com/engine/admin/using_supervisord/) it looks like that could be an option as well as the bash script option (as in my example), that's why I decided to go for the bash script and to ignore supervisor.
And the content of the cron details /workspace/conf/root is this:
* * * * * root echo "Hello world: $(date +%H:%M:%S)" >> /var/log/cron.log 2>&1
(with at the bottom as an empty line \n)
I can not find a way to see that Hello world: $(date +%H:%M:%S) each minute appended to /var/log/cron.log, but to me all the cron/crontab settings are correct.
When I check the logs of the container I can see:
>>> Configuring cron
[ ok ] Starting periodic command scheduler: cron.
>>> Done!
Also, when logging into the running container I can see the cron daemon running:
root#2330ced4daa9:/workspace# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4336 1580 ? Ss+ 13:06 0:00 /bin/sh /workspace/conf/entrypoint.sh
root 14 0.0 0.0 27592 2096 ? Ss 13:06 0:00 /usr/sbin/cron
root 36 0.0 0.0 5956 740 ? S+ 13:06 0:00 tail -f /var/log/cron.log
root 108 0.5 0.1 21948 3692 ? Ss 13:14 0:00 bash
root 114 0.0 0.1 19188 2416 ? R+ 13:14 0:00 ps aux
What am I doing wrong?
Are you sure the Cronjob has execution rights?
chmod 0644 /var/spool/cron/crontabs/root

Xvfb command in docker supervisor conf not working

I have a Docker image based on Ubuntu that runs a supervisor script as the CMD at the end of the Dockerfile. This successfully runs uwsgi and nginx in the container on start up. However, the following appended at the end of the supervisor-app.conf does not work:
[program:Xvfb]
command=/usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
When I open a shell into a running docker instance there is no X instance running:
root#9221694363ea:/# ps aux | grep X
root 39 0.0 0.0 8868 784 ? S+ 15:32 0:00 grep --color=auto X
However, running exactly the same command as in the supervisor-app.conf works
root#9221694363ea:/# /usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
[1] 40
root#9221694363ea:/# ps aux | grep X
root 40 1.2 0.1 170128 21604 ? Sl 15:33 0:00 /usr/bin/Xvfb :1 -screen 0 1024x768x16
root 48 0.0 0.0 8868 792 ? S+ 15:33 0:00 grep --color=auto X
so what's wrong with the line in the supervisor-app.conf?
Supervisor does not handle bash specific operators such as the-run-in-the -background '&' or redirections like '>' as per my original failing config line.
I solved it by using bash -c thus:
[program:Xvfb]
command=bash -c "/usr/bin/Xvfb :1 -screen 0 1024x768x16 &> xvfb.log"
Now when I get into the docker bash shell the Xvfb window is created waiting for me to use it elsewhere in the code.

How can I get my Fortran program to use a certain amount of RAM?

I am trying to write a Fortran program which will eat up a lot of memory (for the reasoning behind this, please see the note at the end of this question). I am doing this by allocating a 3 dimensional array of size (n,n,n) and then deallocating it - continually increasing n until I run out of memory (this should happen when ~16 GB of memory is used). Unfortunately, it seems as if my program is running out of memory long before I see the system resources get up to 16 GB.
Here is my sample code:
1 program fill_mem
2 implicit none
3 integer, parameter :: ikind = selected_int_kind(8)
4 integer, parameter :: rkind = 8
5
6 integer(kind = ikind) :: nfiles = 100
7 integer(kind = ikind) :: n = 1200
8 integer(kind = ikind) :: i, nn
9
10 real(kind = rkind), allocatable :: real_arr(:,:,:)
11
12 character(500) :: sysline
13
14
15 call system('echo ''***no_allocation***'' > outfile')
16 call system('ps aux | grep fill_mem.exe >> outfile')
17 !call system('smem | grep fill_mem.exe >> sm.out')
18 allocate(real_arr(n, n, n))
19
20 nn = 100000
21 do i = 1,nn
22 deallocate(real_arr)
23 n = n + 10
24 print*, 'n = ', n
25 allocate(real_arr(n, n, n))
26 call system('echo ''*************'' >> outfile')
27 write(sysline, *) 'allocation', i, '... n = ', n
28
29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%'
30
31 call system(trim(adjustl('echo '//sysline//'>> outfile')))
32 call system('ps aux | grep fill_mem.exe >> outfile')
33 enddo
34
35 end program fill_mem
and here is the sample output:
1 ***no_allocation***
2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe
3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
5 *************
6 allocation 1 ... n = 1210
7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe
8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
10 *************
11 allocation 2 ... n = 1220
12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe
13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
15 *************
16 allocation 3 ... n = 1230
17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
20 *************
21 allocation 4 ... n = 1240
22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
25 *************
26 allocation 5 ... n = 1250
27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe
30 *************
31 allocation 6 ... n = 1260
32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
35 *************
36 allocation 7 ... n = 1270
37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe
38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile
39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe
Now, I see that the VSZ portion gets up to ~15 GB so I am assuming when I try to address more, it fails with
Operating system error: Cannot allocate memory
Allocation would exceed memory limit
because there is not that much RAM. Why is it that RSS is so far below that, though? When I actually look on my system resources I see about 140 MB being used up (I am running this in a Linux VM and monitoring the system resources through Windows - I have given the GM 16 GB of RAM to use though, so I should see the VM memory increasing until it reaches the 16 GB mark - for what it's worth, the VM has VT-x/Nested Paging/PAE/NX so it should use the physical architecture just like the native OS).
Can anyone explain why I do not see my program actually using up the full 16 GB of RAM and how I can write my code to keep these arrays I am creating in RAM - fully utilizing my available hardware?
NOTE: The reason I am trying to write a sample program which reads a lot of memory is that I am working with data which takes up around 14 GB of space in ascii text. I will need to be working with data A LOT throughout the course of this program, so I want to read it all in at once and then reference it from RAM throughout the duration of the program. To make sure I am doing this correctly, I am trying to write a simple program which will store a very large array (~15 GB) in memory all at once.
(Caveat: The Fortran standard doesn't say how such thing ought to be implemented etc., the description below refers to how Fortran compilers are typically implemented on current operating systems.)
When you execute an ALLOCATE statement (or equivalently, calling malloc() in C, FWIW), you're not actually reserving physical memory, but only mapping address space for your process. That's why the VSZ goes up, but not the RSS. Actually reserving physical memory for your process happens only when you first access the memory (typically at page size granularity, that is, 4 KB on most current hw). So only once you start putting some data into your array does the RSS begin to climb. E.g. a statement like
real_arr = 42.
ought to bump up your RSS to the vicinity of the VSZ.
You probably need to increase the memory allocated to the stack. For example, see http://software.intel.com/en-us/articles/intel-fortran-compiler-increased-stack-usage-of-80-or-higher-compilers-causes-segmentation-fault

Resources