For the following perf record command
perf record -e cycles:k -j any,k -a
I see the following columns when I read it via perf script
nginx 12722 60659.641815: 1 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641828: 1 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641835: 3 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641842: 17 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641848: 97 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641855: 547 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641861: 3060 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641868: 17723 cycles:k: ffffffffa2a72f36 native_write_msr+0x6 (/lib/modules/4.19.119/build/vmlinux
nginx 12722 60659.641884: 99057 cycles:k: ffffffffa2bbdaba irq_work_run_list+0xa (/lib/modules/4.19.119/build/vmlinu
nginx 12722 60659.641939: 274693 cycles:k: ffffffffa329ade4 sock_poll+0x44 (/lib/modules/4.19.119/build/vmlinux)
nginx 12722 60659.642112: 397008 cycles:k: ffffffffa333cc7b tcp_write_xmit+0xbb (/lib/modules/4.19.119/build/vmlinux)
I know that 12722 is the PID. I would like to know what are "60659.641815" and "1 cycles" columns? I guess the former is the time but don't know why it starts from 60659.641815.
You are right, "60659.641815" is representing timestamp in the units of seconds.microseconds. This timestamp represents the number of seconds since the system was started. For more details, read this.
cycles:k represents the event name and the value 1 is not associated with cycles. These values 1,3,17,97, 547 etc. represent the sample period. This means, if you have a sample period of x, at the xth occurence of the event, the hardware counter corresponding to the cycles event overflowed and a sample was written.
A quick and easy check to verify the above statement is to run perf record with a fixed sample period of, say 2, and then see perf script output,
perf record -e cycles:k -j any,k -c 2 -a
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.289 MB perf.data (8 samples) ]
perf script
perf 5746 [000] 54322.047321: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [001] 54322.047350: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [002] 54322.047425: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [003] 54322.047449: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [004] 54322.047473: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [005] 54322.047496: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
swapper 0 [006] 54322.047519: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
MediaPl~back #3 2095 [007] 54322.047545: 2 cycles:k: ffffffffb786ee9a native_write_msr+0xa ([kernel.kallsyms])
Related
I wanted to make use of GCP Cloud Transfer Service to sync data from on-premise. For this I had to install agents and connect them to a pool. I was following the steps here, where they provide docker commands to install a transfer agent. Instead of docker, I used podman:
podman run -ti --name gcloud-config docker.io/google/cloud-sdk gcloud auth application-default login
podman run -d --rm \
--env https_proxy=<some_proxy> \
--volumes-from gcloud-config \
-v <some_dir>:<some_dir> \
gcr.io/cloud-ingest/tsop-agent:latest \
--project-id=<some_project_id> \
--hostname=$(hostname) \
--agent-pool=source_agent_pool
The agents do start but they aren't able to connect to the pool.
If I see the output of the agent container (using podman attach containerID)
0B/s txSum: 0B taskResps[copy:0 delete:0 list:0] ctrlMsgAge:10m50s (??) |
and agent.INFO logs:
Build target: //cloud/transfer/online/onprem/workers/agent:agent
Build id: <some_id>
I1222 06:47:51.288924 3 log_spam.go:51] Command line arguments:
I1222 06:47:51.288926 3 log_spam.go:53] argv[0]: './agent'
I1222 06:47:51.288928 3 log_spam.go:53] argv[1]: '--project-id=<project_id>'
I1222 06:47:51.288930 3 log_spam.go:53] argv[2]: '--hostname=<hostname>'
I1222 06:47:51.288931 3 log_spam.go:53] argv[3]: '--agent-pool=source_agent_pool'
I1222 06:47:51.288933 3 log_spam.go:53] argv[4]: '--container-id=49be0b94bced'
I1222 06:47:51.289408 3 prodlayer.go:217] layer successfully set to NO_LAYER with source DEFAULT
I1222 06:47:53.148699 3 handler.go:45] TaskletHandler initialized to delete at most 1000 objects in parallel:
I1222 06:47:53.148725 3 handler.go:48] TaskletHandler initialized with delete-files: 1024
I1222 06:47:53.148827 3 copy.go:145] TaskletHandler initialized with copy-files: &{0xc00073d2c0 10800000000000}
I1222 06:47:53.148860 3 handler.go:61] TaskletHandler initialized to process at most 256 list outputs in parallel:
I1222 06:48:51.291680 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:49:51.291017 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:50:51.290721 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:51:51.291057 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:52:51.290677 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I1222 06:53:51.290445 3 cpuutilization.go:86] Last minute's CPU utilization: 0
I also checked all the troubleshooting steps here, but couldn't find anything. Is it something to do with using podman instead of docker?
I am able to run containers fine with this combination.
But I noticed - there is no /etc/docker directory on the linux side and when I do ps -eF I get this. I was expecting dockerd and container processes as children of dockerd
rookie#MAIBENBEN-PC:/mnt/c/Users/rookie$ ps -eF
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
root 1 0 0 223 580 6 04:07 ? 00:00:00 /init
root 98 1 0 223 80 5 04:07 ? 00:00:00 /init
root 99 98 0 223 80 5 04:07 ? 00:00:00 /init
rookie 100 99 0 191067 43220 0 04:07 pts/0 00:00:00 docker serve --address unix:///home/rookie/.docker/run/d
root 101 98 0 0 0 1 04:07 ? 00:00:00 [init] <defunct>
root 103 98 0 223 80 7 04:07 ? 00:00:00 /init
root 104 103 0 384463 28888 0 04:07 pts/1 00:00:00 /mnt/wsl/docker-desktop/docker-desktop-proxy --distro-na
root 142 1 0 223 80 4 05:17 ? 00:00:00 /init
root 143 142 0 223 80 6 05:17 ? 00:00:00 /init
rookie 144 143 0 2509 5048 2 05:17 pts/2 00:00:00 -bash
rookie 221 144 0 2654 3264 7 05:21 pts/2 00:00:00 ps -eF
Your Ubuntu session (and all WSL2 sessions) are set up as docker clients, but the actual docker daemon is running in a separate WSL session named "docker-desktop".
I generally recommend leaving this instance alone, as it is auto-configured and managed by Docker Desktop, but if you really want to take a look, run:
wsl -d docker-desktop
... from PowerShell, CMD, or Windows Start/Run.
Note that this instance is running BusyBox, so some commands will be different than you expect. For instance, the -F argument is not valid for ps.
You'll see dockerd and the associated containerd processes here.
There's also a separate image, docker-desktop-data, but it is not bootable (there is no init in it). If you want to see the filesystem, at least, you can wsl --export it and examine the tar file that is created. I wrote up an answer on Super User with details a few months ago.
Thanks for taking the time to read my problem is the following, my auto-escalation policies are associated with a docker container, if the container requires autoscale memonia. In the container the processes (top) our one less load to "docker stats id". There are times when the RAM of the container becomes saturated because the dentry is not live (page cache)
docker stats does not show the actual RAM consumption that the container uses:
docker stats bf257938fa2d 66.54MiB
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O
bf257938fa2d ce88cfdda8f09bc08101 0.00% 66.54MiB / 512MiB 13.00% 1.44MB / 1.26MB 40.3MB / 0B 0
**docker exec -it bf257938fa2d top **
top - 23:24:02 up 53 min, 0 users, load average: 0.01, 0.21, 0.21
Tasks: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 0.3%sy, 0.0%ni, 95.6%id, 0.2%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 15660100k total, 1989516k used, 13670584k free, 95920k buffers
Swap: 0k total, 0k used, 0k free, 1167184k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 11604 2468 2284 S 0.0 0.0 0:00.02 bash
6 root 20 0 309m 12m 7036 S 0.0 0.1 0:00.09 php-fpm
7 root 20 0 59292 7100 6052 S 0.0 0.0 0:00.00 nginx
8 nginx 20 0 59728 4824 3352 S 0.0 0.0 0:00.03 nginx
9 nginx 20 0 59728 4800 3328 S 0.0 0.0 0:00.02 nginx
70 root 20 0 15188 2040 1832 R 0.0 0.0 0:00.02 top
In what way could solve, that RAM consumption is equal in the container (top) and outside the container (docker stats).
Thank you
Scenario:
S3 bucket has 1000 files. I have two machines. Each of these machines has two drives /dev/sda and /dev/sdb.
Constraints: no one single drive can fit all 1000 files. And no one machine can fit all 1000 files.
Desired outcome: distribute 1000 files across 4 drives on two machines using GNU parallel.
I tried things like:
parallel --xapply --joblog out.txt -S:,R echo {1} {2} ::: "/dev/sda" "/dev/sdb" ::: {0..10}
But I get:
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
2 : 1414040436.607 0.037 0 0 0 0 echo /dev/sda 1
4 : 1414040436.615 0.030 0 0 0 0 echo /dev/sda 3
6 : 1414040436.623 0.024 0 0 0 0 echo /dev/sda 5
8 : 1414040436.632 0.015 0 0 0 0 echo /dev/sda 7
10 : 1414040436.640 0.006 0 0 0 0 echo /dev/sda 9
1 R 1414040436.603 0.088 0 0 0 0 echo /dev/sdb 0
3 R 1414040436.611 0.092 0 0 0 0 echo /dev/sdb 2
5 R 1414040436.619 0.095 0 0 0 0 echo /dev/sdb 4
7 R 1414040436.628 0.095 0 0 0 0 echo /dev/sdb 6
9 R 1414040436.636 0.096 0 0 0 0 echo /dev/sdb 8
11 R 1414040436.645 0.094 0 0 0 0 echo /dev/sdb 10
Where 'R' is remote host IP. How do I distribute files (I have all names in a file) from S3 to 4 drives? Thank you.
GNU Parallel is good for starting a new job when an old has finished: It divides the jobs into servers on the fly and not beforehand.
What you are looking for is a way to do this beforehand.
Your --xapply approach seems sound, but you need to force GNU Parallel to distribute evenly to the hosts. Your current approach is dependent on how fast each host finishes, and that will not work in general.
So something like:
parallel echo {1}//{2} ::: sda sdb ::: server1 server2 | parallel --colsep '//' --xapply echo copy {3} to {1} on {2} :::: - filenames.txt
Or:
parallel --xapply echo copy {3} to {1} on {2} ::: sda sda sdb sdb ::: server1 server2 server1 server2 :::: filenames.txt
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
Per the below...supervisord is taking a lot of cpu. Supervisor is managing my uWSGI application. Does that no seem like a lot? How can I reduce this load? Can I change the nice to +1?
1 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 76.6%] 5 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 85.4%]
2 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 85.4%] 6 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 87.2%]
3 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 82.2%] 7 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 72.8%]
4 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||95.3%] 8 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 77.3%]
Mem[||||||||||||||||||||||| 1558/16050MB] Tasks: 57, 15 thr; 12 running
Swp[ 0/7627MB] Load average: 8.75 8.08 7.21
Uptime: 01:51:59
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
29617 root 20 0 491M 144M 2604 R 84.0 0.9 1:13.48 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29615 root 20 0 421M 146M 2616 R 84.0 0.9 1:13.79 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29337 root 20 0 444M 169M 2644 R 83.0 1.1 1:31.70 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29613 root 20 0 423M 148M 2624 R 82.0 0.9 1:16.76 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29333 root 20 0 591M 172M 2604 R 82.0 1.1 1:35.10 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29474 root 20 0 428M 153M 2600 R 81.0 1.0 1:20.31 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
29611 root 20 0 429M 154M 2616 R 80.0 1.0 1:20.54 /usr/local/bin/uwsgi --loop gevent --carbon 103.4.112.50:2003 --socket 127.0.0.1:8060 --processes 7 --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/works
24760 root 20 0 49932 12520 1744 R 33.0 0.1 26:05.37 /usr/bin/python /usr/local/bin/supervisord
Supervisord configuration:
# -*- conf -*-
[include]
files = *.supervisor
[supervisord]
pidfile = /var/run/supervisord.pid
nodaemon=false
[supervisorctl]
serverurl=unix:///var/run/supervisord.sock ; use a unix:// URL for a unix socket
[unix_http_server]
file = /var/run/supervisord.sock
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:uwsgiBidder]
autostart=true
autorestart=true
process_name = uwsgiBidder-%(process_num)s
<% if #enviroment=='production' -%>
command = /usr/local/bin/uwsgi --loop gevent --carbon <%=#carbon_ip_address%>:2003 --socket 127.0.0.1:8060 --processes <%= #bidder_cores %> --pp /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/ --wsgi-file /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/uwsgiBidderServer.py -b 62768 --master --async 2000 --enable-threads --listen 2048 --reload-on-rss 1200 --touch-reload /home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiBidderServer/uwsgiBidderServer.py
<% end -%>
numprocs = 1
numprocs_start = 8060
stopsignal=QUIT
#user=www-data
#--uid www-data
Here is the output of iotop
> Total DISK READ: 0.00 B/s | Total DISK WRITE: 27.46 K/s
> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> 280 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.27 %
> [jbd2/dm-1-8] 24760 be/4 root 0.00 B/s 388.30 K/s 0.00 %
> 0.00 % python /usr/local/bin/supervisord
> 1 be/4 roo
You need to either turn off uwsgi request logging (--disable-logging), or configure uwsgi to log requests elsewhere using --logto or the UDP logging options provided.
What happens now is that your uwsgi processes are sending a lot of data on their stdout, so the kernel is waking up your supervisord process a lot to handle the log data (via select() calls). With 6K qps you really want to have something else handle that volume of log data.