Scenario:
S3 bucket has 1000 files. I have two machines. Each of these machines has two drives /dev/sda and /dev/sdb.
Constraints: no one single drive can fit all 1000 files. And no one machine can fit all 1000 files.
Desired outcome: distribute 1000 files across 4 drives on two machines using GNU parallel.
I tried things like:
parallel --xapply --joblog out.txt -S:,R echo {1} {2} ::: "/dev/sda" "/dev/sdb" ::: {0..10}
But I get:
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
2 : 1414040436.607 0.037 0 0 0 0 echo /dev/sda 1
4 : 1414040436.615 0.030 0 0 0 0 echo /dev/sda 3
6 : 1414040436.623 0.024 0 0 0 0 echo /dev/sda 5
8 : 1414040436.632 0.015 0 0 0 0 echo /dev/sda 7
10 : 1414040436.640 0.006 0 0 0 0 echo /dev/sda 9
1 R 1414040436.603 0.088 0 0 0 0 echo /dev/sdb 0
3 R 1414040436.611 0.092 0 0 0 0 echo /dev/sdb 2
5 R 1414040436.619 0.095 0 0 0 0 echo /dev/sdb 4
7 R 1414040436.628 0.095 0 0 0 0 echo /dev/sdb 6
9 R 1414040436.636 0.096 0 0 0 0 echo /dev/sdb 8
11 R 1414040436.645 0.094 0 0 0 0 echo /dev/sdb 10
Where 'R' is remote host IP. How do I distribute files (I have all names in a file) from S3 to 4 drives? Thank you.
GNU Parallel is good for starting a new job when an old has finished: It divides the jobs into servers on the fly and not beforehand.
What you are looking for is a way to do this beforehand.
Your --xapply approach seems sound, but you need to force GNU Parallel to distribute evenly to the hosts. Your current approach is dependent on how fast each host finishes, and that will not work in general.
So something like:
parallel echo {1}//{2} ::: sda sdb ::: server1 server2 | parallel --colsep '//' --xapply echo copy {3} to {1} on {2} :::: - filenames.txt
Or:
parallel --xapply echo copy {3} to {1} on {2} ::: sda sda sdb sdb ::: server1 server2 server1 server2 :::: filenames.txt
Related
I have an ansible playbook as follows (with irrelevant info stripped):
tasks:
- name: get public_ip4 output
shell: curl http://169.254.169.254/latest/meta-data/public-ipv4
register: public_ip4
- debug: var=public_ipv4.stdout
- name: Create docker_pull
template: <SNIP>
- name: Pull containers
command: "sh /root/pull_agent.sh"
- name: (re)-create the agent
docker_container:
name: agent
image: registry.gitlab.com/project_agent
state: started
exposed_ports: 8889
published_ports: 8889:8889
recreate: yes
env:
host_machine: public_ipv4.stdout
The target machine is an AWS EC2 instance. The purpose is to get its public IPv4 address and give it as an environment variable to a container. The container has a Python instance, Agent, that will use os.environ.get(host_machine) to thus access the IPv4 address of the EC2 instance.
The output from the debug logs is (with irrelevant info removed and the ipv4 address replaced with ):
PLAY [swarm_manager_main] ******************************************************
TASK [get public_ip4 output] ***************************************************
[WARNING]: Consider using the get_url or uri module rather than running
'curl'. If you need to use command because get_url or uri is insufficient you
can add 'warn: false' to this command task or set 'command_warnings=False' in
ansible.cfg to get rid of this message.
changed: [tm001.stackl.io] => {"changed": true, "cmd": "curl http://169.254.169.254/latest/meta-data/public-ipv4", "delta": "0:00:00.013260", "end": "2019-08-02 08:36:26.649844", "rc": 0, "start": "2019-08-02 08:36:26.636584", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 12 100 12 0 0 2600 0 --:--:-- --:--:-- --:--:-- 3000", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 12 100 12 0 0 2600 0 --:--:-- --:--:-- --:--:-- 3000"], "stdout": "<HIDDEN>", "stdout_lines": ["52.210.80.33"]}
TASK [debug] *******************************************************************
ok: [tm001.stackl.io] => {
"public_ipv4.stdout": "VARIABLE IS NOT DEFINED!: 'public_ipv4' is undefined"
}
TASK [Create docker_pull] ******************************************************
<SNIP>
TASK [Pull containers] *********************************************************
<SNIP>
TASK [(re)-create the agent] ********************************************
changed: <SNIP> ["host_machine=public_ipv4.stdout", <SNIP>
I don't understand why the public_ipv4 variable is not used correctly. I've tried multiple things (including setting set_fact or setting a new variable) but to no avail.
What am I doing wrong?
There is a typo in your playbook: ip4 and ipv4
register: public_ip4
debug: var=public_ipv4.stdout
Thanks for taking the time to read my problem is the following, my auto-escalation policies are associated with a docker container, if the container requires autoscale memonia. In the container the processes (top) our one less load to "docker stats id". There are times when the RAM of the container becomes saturated because the dentry is not live (page cache)
docker stats does not show the actual RAM consumption that the container uses:
docker stats bf257938fa2d 66.54MiB
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O
bf257938fa2d ce88cfdda8f09bc08101 0.00% 66.54MiB / 512MiB 13.00% 1.44MB / 1.26MB 40.3MB / 0B 0
**docker exec -it bf257938fa2d top **
top - 23:24:02 up 53 min, 0 users, load average: 0.01, 0.21, 0.21
Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 0.3%sy, 0.0%ni, 95.6%id, 0.2%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 15660100k total, 1989516k used, 13670584k free, 95920k buffers
Swap: 0k total, 0k used, 0k free, 1167184k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 11604 2468 2284 S 0.0 0.0 0:00.02 bash
6 root 20 0 309m 12m 7036 S 0.0 0.1 0:00.09 php-fpm
7 root 20 0 59292 7100 6052 S 0.0 0.0 0:00.00 nginx
8 nginx 20 0 59728 4824 3352 S 0.0 0.0 0:00.03 nginx
9 nginx 20 0 59728 4800 3328 S 0.0 0.0 0:00.02 nginx
70 root 20 0 15188 2040 1832 R 0.0 0.0 0:00.02 top
In what way could solve, that RAM consumption is equal in the container (top) and outside the container (docker stats).
Thank you
I want to have multiple instances of an aerospike-server node so load balance because I hit the server with so many requests I get an error saying 'Error Code 22: Operation not allow at this time'.
I assume that having more aerospike-server nodes that i can lessen the load on one node and therefore complete more operations? I'm jsut not sure where to start with this.
I need these nodes all locally on one host. Any help would be great.
My network knowledge isn't great so forgive me for any
My DockerFile:
FROM aerospike/aerospike-server
MAINTAINER "xxxx"
ADD aerospike.conf /etc/aerospike/
docker-compose.yml
version: "3"
services:
aerospike-server:
image: amc
container_name: aerospike-server
ports:
- "3000:3000"
- "3001:3001"
- "3002:3002"
- "3003:3003"
aerospike-amc:
image: mrbar42/aerospike-amc
container_name: aerospike-amc
ports:
- "8081:8081"
aerospike.conf network stanza
network {
service {
address any
port 3000
# Uncomment the following to set the `access-address` parameter to the
# IP address of the Docker host. This will the allow the server to correctly
# publish the address which applications and other nodes in the cluster to
# use when addressing this node.
# access-address <IPADDR>
}
heartbeat {
# mesh is used for environments that do not support multicast
mode mesh
port 3002
# use asinfo -v 'tip:host=<ADDR>;port=3002' to inform cluster of
# other mesh nodes
interval 150
timeout 10
}
fabric {
# address any
port 3001
}
info {
# address any
port 3003
}
}
You could use a multicast heartbeat configuration and get a small cluster of aerospike nodes running on a single machine.
Here are the steps I used:
1) Get the latest Dockerfile from the aerospike repo
git clone https://github.com/aerospike/aerospike-server.docker.git
2) Modify the heartbeat section and use multicast:
heartbeat {
mode multicast
multicast-group 239.1.99.2
port 9918
interval 150
timeout 10
}
3) Build image and cluster with either docker-compose or custom script.
I used the following custom makecluster.sh bash shell script:
cat makecluster.sh
docker build -t "aerospike/aerospike-server" .
for i in {3..9} ; do
echo $i;
docker stop aerospike${i}
docker rm aerospike${i}
docker run -tid --name aerospike${i} -p ${i}000:3000 -p ${i}001:3001 -p ${i}002:3002 -p ${i}003:3003 aerospike/aerospike-server
done;
and run:
./makecluster.sh
Sending build context to Docker daemon 12.8 kB
Step 1/10 : FROM ubuntu:xenial
---> 2fa927b5cdd3
Step 2/10 : ENV AEROSPIKE_VERSION 3.12.1
---> Using cache
---> f9740fe0e5b5
Step 3/10 : ENV AEROSPIKE_SHA256 16ceb0d131892c3caab31ee031f7e37d7a611c22686fec2692facde87ac69a00
---> Using cache
---> 5e7fbc820f7a
Step 4/10 : RUN apt-get update -y && apt-get install -y wget python python-argparse python-bcrypt python-openssl logrotate net-tools iproute2 iputils-ping && wget "https://www.aerospike.com/artifacts/aerospike-server-community/${AEROSPIKE_VERSION}/aerospike-server-community-${AEROSPIKE_VERSION}-ubuntu16.04.tgz" -O aerospike-server.tgz && echo "$AEROSPIKE_SHA256 *aerospike-server.tgz" | sha256sum -c - && mkdir aerospike && tar xzf aerospike-server.tgz --strip-components=1 -C aerospike && dpkg -i aerospike/aerospike-server-*.deb && dpkg -i aerospike/aerospike-tools-*.deb && mkdir -p /var/log/aerospike/ && mkdir -p /var/run/aerospike/ && rm -rf aerospike-server.tgz aerospike /var/lib/apt/lists/* && dpkg -r wget ca-certificates && dpkg --purge wget ca-certificates && apt-get purge -y
---> Using cache
---> a1d797e088fa
Step 5/10 : COPY aerospike.conf /etc/aerospike/aerospike.conf
---> Using cache
---> 7bb9f5326c7b
Step 6/10 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> 345534f6c3b8
Step 7/10 : VOLUME /opt/aerospike/data
---> Using cache
---> 3f07a5e4b113
Step 8/10 : EXPOSE 3000 3001 3002 3003
---> Using cache
---> e081dff6a725
Step 9/10 : ENTRYPOINT /entrypoint.sh
---> Using cache
---> 923ff0299e70
Step 10/10 : CMD asd
---> Using cache
---> a05c71468897
Successfully built a05c71468897
3
aerospike3
aerospike3
4d2df0569581c967d29750a5f7f6e67fc1835e1f6b78157eb5c8b11f843a222b
4
aerospike4
aerospike4
3173f06d9012996ce318855d982fbc20c5985beb7e356863a0075ce3ae337b21
5
aerospike5
aerospike5
10592f8ca508c678cd67695402f3c23ad417716772cf5a20de91c3e178bb1c28
6
aerospike6
aerospike6
7ddac9e31db5b618114c155b40d8ad524b629e96c514837a36854cf420a29d5c
7
aerospike7
aerospike7
6168421deffead460ac238b2a9817769e29d3c68eca33863cf9b0e19a8c09a0f
8
aerospike8
aerospike8
9d1761592c30b1460c064ef06f70067217e1b1c61f3efc2b016113b4869cfc6f
9
aerospike9
aerospike9
67c37f2f6ebae065eef978dcf4683b6ad5bd18bed0411e3d6681c723776c1834
5) Verify cluster info :
Use docker exec to run asadm tool
docker exec -ti aerospike7 asadm
Aerospike Interactive Shell, version 0.1.9
Found 7 nodes
Online: 172.17.0.2:3000, 172.17.0.8:3000, 172.17.0.3:3000, 172.17.0.4:3000, 172.17.0.7:3000, 172.17.0.6:3000, 172.17.0.5:3000
Admin>
Admin> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Rackaware Client Uptime
. Id . . Size Key Integrity . Mode Conns .
172.17.0.2:3000 BB9020011AC4202 172.17.0.2:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 7 00:06:03
172.17.0.3:3000 BB9030011AC4202 172.17.0.3:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 8 00:06:02
172.17.0.4:3000 BB9040011AC4202 172.17.0.4:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 8 00:06:00
172.17.0.5:3000 BB9050011AC4202 172.17.0.5:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 6 00:05:58
172.17.0.7:3000 BB9070011AC4202 172.17.0.7:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 10 00:05:55
172.17.0.8:3000 *BB9080011AC4202 172.17.0.8:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 6 00:05:52
6168421deffe:3000 BB9060011AC4202 172.17.0.6:3000 C-3.12.1 7 8DBD43898C508D41 True BB9080011AC4202 none 8 00:05:56
Number of rows: 7
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop
. . . . (Objects,Tombstones) (Objects,Tombstones) Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%
. . . . . . . . (tx,rx) . . . . . . .
test 172.17.0.2:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 172.17.0.3:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 172.17.0.4:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 172.17.0.5:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 172.17.0.7:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 172.17.0.8:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 6168421deffe:3000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) (0.000 ,0.000 ) 0.000 B 0.000 B
Number of rows: 8
Note: Tools like AMC or asadm , may not be able to see the full cluster if running outside of a docker container on the host machine. In such use case, you may need to try a docker host network and different listening ports in aerospike.conf or the alternative is to run AMC in a container.
6) If using --network=host then you will need to modify the listening ports in aerospike.conf. Each instance running on the same host with same ip address should use a different port when using host network.
Example:
cat makecluster.sh
docker build -t "aerospike/aerospike-server" .
for i in {6..9} ; do
echo $i;
docker stop aerospike${i}
docker rm aerospike${i}
docker run -tid --name aerospike${i} -v /root/docker/aerospike-server.docker/aerospike${i}:/etc/aerospike --network host -p ${i}000:${i}000 -p ${i}001:${i}001 -p ${i}002:${i}002 -p ${i}003:${i}003 aerospike/aerospike-server
done;
Sample config directory and file:
cat aerospike7/aerospike.conf
# Aerospike database configuration file.
# This stanza must come first.
service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 4
transaction-queues 4
transaction-threads-per-queue 4
proto-fd-max 15000
}
logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
context any info
}
# Send log messages to stdout
console {
context any info
}
}
network {
service {
# Add interface name for network card used by host
address br0
# Modify port to be used by your aerospike container
port 7000
# Uncomment the following to set the `access-address` parameter to the
# IP address of the Docker host. This will the allow the server to correctly
# publish the address which applications and other nodes in the cluster to
# use when addressing this node.
# access-address <IPADDR>
}
heartbeat {
mode multicast
multicast-group 239.1.99.2
port 9918
interval 150
timeout 10
}
fabric {
# Modify port to be used by your aerospike container
port 7001
}
info {
# Modify port to be used by your aerospike container
port 7003
}
}
namespace test {
replication-factor 2
memory-size 1G
default-ttl 5d # 5 days, use 0 to never expire/evict.
# storage-engine memory
# To use file storage backing, comment out the line above and use the
# following lines instead.
storage-engine device {
file /opt/aerospike/data/test.dat
filesize 4G
data-in-memory true # Store data in memory in addition to file.
}
}
asadm output:
asadm -p 7000
Aerospike Interactive Shell, version 0.1.9
Found 4 nodes
Online: 192.168.100.148:9000, 192.168.100.148:7000, 192.168.100.148:8000, 192.168.100.148:6000
Admin> info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Ip Build Cluster Cluster Cluster Principal Rackaware Client Uptime
. Id . . Size Key Integrity . Mode Conns .
node.local:6000 177160DE04CA0568 192.168.120.148:6000 C-3.12.1 4 5009B1D835FA3C06 True 232960DE04CA0568 none 3 00:28:37
node.local:7000 1B5960DE04CA0568 192.168.120.148:7000 C-3.12.1 4 5009B1D835FA3C06 True 232960DE04CA0568 none 3 00:28:34
node.local:8000 1F4160DE04CA0568 192.168.120.148:8000 C-3.12.1 4 5009B1D835FA3C06 True 232960DE04CA0568 none 3 00:28:30
node.local:9000 *232960DE04CA0568 192.168.120.148:9000 C-3.12.1 4 5009B1D835FA3C06 True 232960DE04CA0568 none 4 00:28:25
Number of rows: 4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace Node Avail% Evictions Master Replica Repl Stop Pending Disk Disk HWM Mem Mem HWM Stop
. . . . (Objects,Tombstones) (Objects,Tombstones) Factor Writes Migrates Used Used% Disk% Used Used% Mem% Writes%
. . . . . . . . (tx,rx) . . . . . . .
test node.local:6000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test node.local:7000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test node.local:8000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test node.local:9000 99 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) 2 false (0.000 ,0.000 ) 0.000 B 0 50 0.000 B 0 60 90
test 0.000 (0.000 ,0.000 ) (0.000 ,0.000 ) (0.000 ,0.000 ) 0.000 B 0.000 B
I am running the docker release of openFOAM. While running openFOAM, I can't access any of the volumes that I have set up in /mnt. I can see them when I run:
bash-4.1$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 29.8G 0 disk
|-sda1 8:1 0 200M 0 part
|-sda2 8:2 0 500M 0 part
`-sda3 8:3 0 29.1G 0 part
`-luks-c551009c-5ab5-4526-85fa-45105a445734 (dm-0)
253:0 0 29.1G 0 crypt
|-korora_a00387863--6-root (dm-1) 253:1 0 26.1G 0 lvm /etc/passwd
`-korora_a00387863--6-swap (dm-2) 253:2 0 3G 0 lvm
sdb 8:16 0 465.8G 0 disk
|-sdb1 8:17 0 137.9G 0 part
|-sdb2 8:18 0 158.7G 0 part
`-sdb3 8:19 0 169.2G 0 part
sdg 8:96 1 15G 0 disk
loop0 7:0 0 100G 0 loop
`-docker-253:1-265037-pool (dm-3) 253:3 0 100G 0 dm
`-docker-253:1-265037-10f82f41512f788ec85215e8764cd3c5b0973d548fe4db2fcbcbaf50db6a4b9c (dm-4)
253:4 0 10G 0 dm /
loop1 7:1 0 2G 0 loop
`-docker-253:1-265037-pool (dm-3) 253:3 0 100G 0 dm
`-docker-253:1-265037-10f82f41512f788ec85215e8764cd3c5b0973d548fe4db2fcbcbaf50db6a4b9c (dm-4)
253:4 0 10G 0 dm /
However, none of these show up in /dev, so I don't know how to mount the volumes that I want. It seems like there is a better solution than manually mounting the volume each time I use openFOAM. Any ideas would be welcome, I don't understand the docker documentation.
You haven't show us exactly what you mean by "volumes set up in /mnt", so there will be a lot of guesswork in this answer w/r/t what you're actually trying to do.
If you are trying to mount block devices on your host and make them available in your container, the normally way you would go about this is:
Mount the device somewhere on your host (e.g., in /mnt)
Use the -v argument to docker run to expose that mountpoint inside a container, as in:
docker run -v /mnt/volume1:/volume1 alpine sh
The above command line would expose /mnt/volume1 on the host as /volume1 inside the container.
If you find that you are often running the same container with the same set of volumes, and you're tired of long command lines, just drop the docker run command into a shell script, or consider using something like docker-compose to help automate things.
I have a slurm job like this:
#!/bin/bash
#SBATCH -o %A.%N.out
#SBATCH -e %A.%N.err
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH -n 16
#SBATCH --export=ALL
#SBATCH -t 1:00:00
cmd1 input1 > o1
cmd2 o1 > o2
cmd3 o2 > o3
With sacct, one can get the time and cpu usage for the whole job. I am also interested to get those info for cmd1 and cmd3 specifically. How can you do that? Will job step and srun help do that?
You can get a separate entry on sacct per step.
If you run your commands with srun they will generate a step and each one will be monitored and have its own entry.
After this you will see in the sacct output one line for the whole job, one for the batch step, and one for each of the steps on the script (srun/mpirun commands)
You can use time -v to get advanced information about timing and resources used. Not that this refers to the binary /usr/bin/time, not the shell built-in time:
$ /usr/bin/time -v ls /
bin dev home lib64 media opt root sbin sys usr
boot etc lib lost+found mnt proc run srv tmp var
Command being timed: "ls /"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2136
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 126
Voluntary context switches: 1
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
You can prepend this to any command in your batch script.