I have installed jenkins version 1.614 on ubuntu 12.04 with configuration - 32 GB ram, 2 TB hdd and 8 CPU cores. Currently Jenkins has 594 jobs added.
In normal condition when no job is running cpu usage is 0% but When I start any job build cpu usage suddenly reaches 700-800%.
Following are the stats for cpu usage.
top - 06:29:55 up 160 days, 17:43, 3 users, load average: 4.27, 2.54, 2.43
Tasks: 123 total, 2 running, 118 sleeping, 0 stopped, 3 zombie
Cpu0 : 96.7%us, 0.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 95.7%us, 1.0%sy, 0.0%ni, 3.0%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 96.7%us, 0.7%sy, 0.0%ni, 2.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 92.0%us, 0.7%sy, 0.0%ni, 7.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 96.7%us, 0.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 96.7%us, 0.7%sy, 0.0%ni, 2.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 97.0%us, 0.0%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32809732k total, 26551884k used, 6257848k free, 481144k buffers
Swap: 16768892k total, 8376k used, 16760516k free, 15064772k cached
3852 jenkins 20 0 12.5g 8.1g 21m S 774 26.0 2095:06 java
14 root 20 0 0 0 0 S 0 0.0 5:05.81 kworker/2:0
67 root 20 0 0 0 0 S 0 0.0 4:51.63 kworker/3:1
399 root 20 0 0 0 0 S 0 0.0 4:12.45 jbd2/md2-8
1251 root 20 0 0 0 0 S 0 0.0 6:02.19 flush-9:2
6754 appster 20 0 1052m 152m 2244 S 0 0.5 317:39.39 statsd
1 root 20 0 24196 1528 832 S 0 0.0 0:18.22 init
I have also deleted job builds older than 30 days and current running job build is also small job, Still cpu usage is very high.
Related
cat /proc/net/sctp/assocs
ASSOC-ID ASSOC SOCK STY SST ST HBKT TX_QUEUE RX_QUEUE UID INODE LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC wmema wmemq sndbuf rcvbuf
13 ffff8800b93a9000 ffff8800ac752300 2 1 3 9176 0 0 0 20735 3905 48538 *192.168.44.228 <-> *A172.16.236.92 7500 10 10 2 0 0 23 1 0 2000000 2000000
0 ffff88042aea3000 ffff880422e88000 0 10 1 40840 0 0 0 9542 3868 3868 *10.127.58.66 <-> *U10.127.115.194 7500 17 17 10 0 0 0 1 0 8388608 8388608
My Erlang applicaation processed crashed and then exited, but found that the erlang VM is still running.
I could recieve pong when ping this "suspended node"
Types regs() and the results show below, there is not my app process.
(hub#192.168.1.140)4> regs().
** Registered procs on node 'hub#192.168.1.140' **
Name Pid Initial Call Reds Msgs
application_controlle <0.7.0> erlang:apply/2 30258442 1390
auth <0.20.0> auth:init/1 189 0
code_server <0.26.0> erlang:apply/2 1194028 0
erl_epmd <0.19.0> erl_epmd:init/1 138 0
erl_prim_loader <0.3.0> erlang:apply/2 2914236 0
error_logger <0.6.0> gen_event:init_it/6 49983527 0
file_server_2 <0.25.0> file_server:init/1 16185407 0
global_group <0.24.0> global_group:init/1 107 0
global_name_server <0.13.0> global:init/1 1385 0
gr_counter_sup <0.43.0> supervisor:gr_counter_sup 253 0
gr_lager_default_trac <0.70.0> gr_counter:init/1 121 0
gr_lager_default_trac <0.72.0> gr_manager:init/1 46 0
gr_lager_default_trac <0.69.0> gr_param:init/1 117 0
gr_lager_default_trac <0.71.0> gr_manager:init/1 46 0
gr_manager_sup <0.45.0> supervisor:gr_manager_sup 484 0
gr_param_sup <0.44.0> supervisor:gr_param_sup/1 253 0
gr_sup <0.42.0> supervisor:gr_sup/1 237 0
inet_db <0.16.0> inet_db:init/1 749 0
inet_gethost_native <0.176.0> inet_gethost_native:serve 4698517 0
inet_gethost_native_s <0.175.0> supervisor_bridge:inet_ge 41 0
init <0.0.0> otp_ring0:start/2 30799457 0
kernel_safe_sup <0.35.0> supervisor:kernel/1 278 0
kernel_sup <0.11.0> supervisor:kernel/1 47618 0
lager_crash_log <0.52.0> lager_crash_log:init/1 97712230 0
lager_event <0.50.0> gen_event:init_it/6 1813660437 0
lager_handler_watcher <0.51.0> supervisor:lager_handler_ 358 0
lager_sup <0.49.0> supervisor:lager_sup/1 327 0
net_kernel <0.21.0> net_kernel:init/1 110769667 0
net_sup <0.18.0> supervisor:erl_distributi 313 0
os_cmd_port_creator <0.582.0> erlang:apply/2 81 0
rex <0.12.0> rpc:init/1 15653480 0
standard_error <0.28.0> erlang:apply/2 9 0
standard_error_sup <0.27.0> supervisor_bridge:standar 41 0
timer_server <0.100.0> timer:init/1 59356077 0
user <0.31.0> group:server/3 23837008 0
user_drv <0.30.0> user_drv:server/2 12239455 0
** Registered ports on node 'hub#192.168.1.140' **
Name Id Command
ok
It rarely occurs, but anyone explains it?
System: CentOS 5.8
Erlang: R15B03
I am experiencing out-of-memory crashed with my app on iOS on both iPhone 6 and 5s. I am sure that memory is the problem, because I receive memory warnings and also because of the crash log.
In the app I do video processing and process around 400 images in 10 seconds for one post-processing step. Each frame's resources get properly released in an autorelease pool. The crashes come after a random amount of repetition.
I allocate a lot of memory during those 10 seconds (a few hundred MB) in total, but the memory gets released after each processed image, due to the custom autorelease pool.
However, when I try to profile the root of the problem, I can not even detect high memory consumption, let alone search for the source of the problem. I have analyzed the used physical memory in the activity monitor in Instruments and neither my app nor mediaserverd consume more than 30 MB at any given moment.
Why do I get memory crashes, when my app is only using 30MB of memory at peak? How can I measure and debug the memory consumption correctly?
Any pointers a greatly appreciated!
Here is a screenshot of the memory report at runtime:
Here is the crash log:
Incident Identifier: 2B5E9345-964D-4F6E-87D2-5517F0C22531
CrashReporter Key: 2033c0b344ea7cbf4f153cd862d1e7b68969b42e
Hardware Model: iPhone6,2
OS Version: iPhone OS 8.1.1 (12B435)
Kernel Version: Darwin Kernel Version 14.0.0: Mon Nov 3 22:23:57 PST 2014; root:xnu-2783.3.22~1/RELEASE_ARM64_S5L8960X
Date: 2015-01-26 12:04:35 +0100
Time since snapshot: 672 ms
Free pages: 2515
Active pages: 23951
Inactive pages: 11816
Speculative pages: 643
Throttled pages: 0
Purgeable pages: 0
Wired pages: 133367
File-backed pages: 10813
Anonymous pages: 25597
Compressions: 8823352
Decompressions: 752051
Compressor Size: 83913
Uncompressed Pages in Compressor: 155021
Page Size: 16384
Largest process: RedGlam
Processes
Name | <UUID> | CPU Time| rpages| purgeable| recent_max| lifetime_max| fds | [reason] | (state)
timed <6fa98ab7f5de312b9bfed47e04e3a43e> 0.075 240 0 +5 701 50 [vm-pageshortage] (daemon) (idle)
calaccessd <0a7ad7bbfb523bfdbae43aa6f21279f6> 0.134 462 0 +47 1279 50 [vm-pageshortage] (daemon) (idle)
MobileGestaltHel <19968f31a89230a6b66e51ad668f0921> 0.028 163 0 - 522 50 [vm-pageshortage] (daemon) (idle)
awdd <58036e1703903ee798a8803de204c300> 0.123 405 0 - 1076 50 [vm-pageshortage] (daemon) (idle)
WirelessRadioMan <c4181e6d863133e8aa0c95e77a7bb206> 0.051 293 0 - 787 50 [vm-pageshortage] (daemon) (idle)
notification_pro <b143453e80393938a7ba23a0181dc52c> 0.158 148 0 - 451 50 [vm-pageshortage] (daemon)
com.apple.dt.ins <c2263ead004b339f9fe48bbdf95286c9> 30.091 255 0 - 787 50 [vm-pageshortage] (daemon)
debugserver <f00a5e2ac8f73e7e893c711d88c344a6> 24.790 208 0 - 681 50 [vm-pageshortage] (daemon)
MobileMail <4b48abd990e93dbea47db1cbf328da9e> 6.030 1518 0 - 4299 50 (resume) (continuous)
lsd <f554bd07b90a3cfc9d9ef9f8e234833c> 1.443 333 0 - 859 50 (daemon)
tccd <f2878273872231afa1a6e0af2dcb73a6> 0.619 278 0 - 861 50 (daemon)
MYAPP <417ba797cf3e39df82d79baf41050692> 389.444 105870 0 - 13874 50 (audio) (frontmost) (resume)
ptpd <a06176d3eefe3e3c8549bb4f6d340658> 1.913 817 0 - 2043 50 (daemon)
BTServer <2d0fc0974c073a0aafc9954110080950> 16.246 572 0 - 1859 50 (daemon)
wifid <43e56e539a6a3114bf4cd7646c8dd90e> 149.958 561 0 - 1564 50 (daemon)
discoveryd <68f73878299336d7872b0ae9ce3f7f08> 179.311 585 0 - 1220 100 (daemon)
locationd <5b826e2c09c23eaaa2acc2472269cb30> 4461.461 2209 0 - 5025 50 (daemon)
lockdownd <3a0b3375ad6e391da37a1f79f46843b0> 39.278 364 0 - 1345 50 (daemon)
syslogd <05f6b5e5512938a892bac5af23ab1c08> 127.902 240 0 - 818 50 (daemon)
powerd <2b4ae8758a5b3b709a97c452ec08923b> 56.978 310 0 - 579 50 (daemon)
imagent <d5e037ad2173362d8a6077788b2d7074> 13.323 529 0 - 1354 50 (daemon)
identityservices <9d4b00e3c6003685ac8697c59f4e4d38> 16.729 687 0 - 1794 50 (daemon)
vmd <c3b3270d187f3dcaa843ba73f01ff8cb> 0.482 595 0 - 2598 50 (daemon)
cfprefsd <4325eab208063b998046460a4c2ee484> 32.394 449 0 - 742 50 (daemon)
iaptransportd <4bf77076d69630e389ba64229c526723> 20.807 364 0 - 971 50 (daemon)
apsd <bb925404cb1137b09b85671a8d2c7656> 62.713 738 0 - 1892 50 (daemon)
networkd <fa2acedf0b0035269d66a72e28c3a95a> 307.451 716 0 - 1585 50 (daemon)
sharingd <1ed17c64831f32ea9cbb47e48c4d222c> 29.442 944 0 - 2298 50 (daemon)
dataaccessd <33bcaea3bc473f128685f4df14a115eb> 13.535 729 0 - 2230 50 (daemon)
SCHelper <779250b8a48638958a5922f6ae1066fa> 10.160 134 0 - 324 50 (daemon)
searchd <e5c5e5675c0935eaab5feb15ebc0b934> 5.392 888 0 - 2986 50 (daemon)
mediaserverd <a0354e528bc431958df0d50830bead36> 1080.882 91747 0 - 21944 200 (daemon)
syslog_relay <087c67d0371b324fb6df48442016ec90> 6.746 114 0 - 237 50 (daemon)
SpringBoard <96f929dd23123d8bbc9ba2a0bb48bde1> 1108.457 8031 0 - 17537 50
backboardd <e263837653b434f1880f9d37b3926998> 7219.545 7865 0 - 4676 50 (daemon)
UserEventAgent <f5a211b9c88e3fa481f2bd1ee1f5a921> 1084.316 1000 0 - 2811 100 (daemon)
fseventsd <16c9b62bb28c388ca10d54dbff18c4f8> 20.426 496 0 - 843 50 (daemon)
configd <ed40fcde35ae337ab3b70073199564b1> 117.075 537 0 - 1463 50 (daemon)
fairplayd.H2 <cae337642f6d396b82ac54e72bc0e0a4> 6.550 151 0 - 1433 50 (daemon)
wirelessproxd <ab1fa7e43a7c3f9393533404c2cc80b8> 1.814 264 0 - 986 50 (daemon)
assertiond <10ec04add18f3ecd8a8efbb1cc4e2bd6> 18.292 325 0 - 1045 50 (daemon)
aggregated <281958649a3130aab6ecb1aa47f0a6c1> 2273.629 1367 0 - 2822 50 (daemon)
distnoted <cb5e76091dc53ceeaf65290f8e197a89> 4.937 207 0 - 335 50 (daemon)
discoveryd_helpe <492c39ae2d643adca0ed971675c77406> 0.120 156 0 - 752 50 (daemon)
filecoordination <5ec159db1afe3317878b8ab794e2d7d1> 1.259 308 0 - 941 50 (daemon)
DTMobileIS <2fdc94aa5069338e815fbe3a13e3d95c> 551.592 2538 0 - 36844 50 (daemon)
gputoolsd <97e54c888a9a3978a85fd965a71e7669> 9.503 1031 0 - 2910 50 (daemon)
CommCenter <33412ab229c738c8860c70803fed173b> 787.039 1465 0 - 4737 50 (daemon)
notifyd <5fa8fd5e44c83f64be1475b882b16c82> 142.939 384 0 - 441 50 (daemon)
ReportCrash <e946799f25f833fd9b37a6a1c7b1993c> 0.039 166 0 - 551 50 (daemon)
**End**
Here is the code for one of the effects I am using (renderAnimation2DOverFace does not allocate any memory):
- (CIImage *) render:(CIImage*)targetImage imageContext:(CIContext*) imageContext
facialFeatures:(NSArray*)facialFeatures currentFrameInd:(int)frameInd
{
if ( !facialFeatures ) return targetImage;
#autoreleasepool {
UIImage *editingImage = [[UIImage alloc] initWithCIImage:targetImage];
UIGraphicsBeginImageContextWithOptions(editingImage.size, NO, 1.0);
[editingImage drawInRect:CGRectMake( 0, 0, editingImage.size.width, editingImage.size.height)];
for ( ArtechFacialFeature *facialFeature in facialFeatures ) {
[self renderAnimation2DOverFace:facialFeature.bounds
currentFrameInd:frameInd];
}
UIImage *resultImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
targetImage = [targetImage initWithCGImage:resultImage.CGImage options:nil];
editingImage = nil;
resultImage = nil;
}
return targetImage;
}
This is another video effect routine:
- (CIImage *) render:(CIImage*)targetImage imageContext:(CIContext*) imageContext
facialFeatures:(NSArray*)facialFeatures currentFrameInd:(int)frameInd
{
if ( !facialFeatures ) return targetImage;
#autoreleasepool {
CGImageRef inputImage = [imageContext createCGImage:targetImage fromRect:[targetImage extent]];
GPUImagePicture *sourcePicture = [[GPUImagePicture alloc] initWithCGImage:inputImage];
GPUImageOutput *currentOutput = [self createFilterChain:sourcePicture facialFeatures:facialFeatures
imageExtent:targetImage.extent];
[currentOutput useNextFrameForImageCapture];
[sourcePicture processImage];
UIImage *currentFilteredVideoFrame = [currentOutput imageFromCurrentFramebuffer];
targetImage = [targetImage initWithCGImage:currentFilteredVideoFrame.CGImage];
currentFilteredVideoFrame = nil;
[sourcePicture removeAllTargets];
sourcePicture = nil;
[currentOutput removeOutputFramebuffer];
currentOutput = nil;
CFRelease( inputImage );
[[GPUImageContext sharedFramebufferCache] purgeAllUnassignedFramebuffers];
}
return targetImage;
}
When in a loop doing processing that uses autoreleased memory you need to put an autorelease pool inside the loop. Note that many Cocoa and Foundation methods use auto released memory.
The problem is that each operation is retaining and auto-releasing memory but the memory is not actually released until the current autoreleasepool is drained, usually in the run loop and that is not being called because of the tight loop.
Example:
for (...) {
#autoreleasepool {
perform work
}
}
You can put an autoreleasepool around smaller parts of the operations.
To debug reduce the loop count so you don't crash and then use Heapshot:
Use instruments to check for leaks and memory loss due to retained but not leaked memory. The latter is unused memory that is still pointed to. Use Mark Generation (Heapshot) in the Allocations instrument on Instruments.
For HowTo use Heapshot to find memory creap, see: bbum blog
Basically the method is to run Instruments allocate tool, take a heapshot, run an iteration of your code and take another heapshot repeating 3 or 4 times. This will indicate memory that is allocated and not released during the iterations.
To figure out the results disclose to see the individual allocations.
If you need to see where retains, releases and autoreleases occur for an object use instruments:
Run in instruments, in Allocations set "Record reference counts" on (For Xcode 5 and lower you have to stop recording to set the option). Cause the app to run, stop recording, drill down and you will be able to see where all retains, releases and autoreleases occurred.
The problem was that there was a CIImage which was de-referenced outside of the autoreleasepool and outside of the render methods, which I have posted in the question.
It is the targetImage parameter that is being passed as an argument. So now I added another autorelease pool outside of the method call and that fixed all my memory problems.
It still is a mystery to me though why those unreleased allocations of the CIImage did not manifest in the amount of memory used in the memory monitor and other tools.
Also, I do not understand, why the crashed would only happen occasionally.
Using the Images package, I can open up a color image, convert it to Gray scale and then :
using Images
img_gld = imread("...path to some color jpg...")
img_gld_gs = convert(Image{Gray},img_gld)
#change from floats to Array of values between 0 and 255:
img_gld_gs = reinterpret(Uint8,data(img_gld_gs))
Now I've got a 1920X1080 array of Uint8's:
julia> img_gld_gs
1920x1080 Array{Uint8,2}
Now I want to get a histogram of the 2D array of Uint8 values:
julia> hist(img_gld_gs)
(0.0:50.0:300.0,
6x1080 Array{Int64,2}:
1302 1288 1293 1302 1297 1300 1257 1234 … 12 13 13 12 13 15 14
618 632 627 618 623 620 663 686 189 187 187 188 185 183 183
0 0 0 0 0 0 0 0 9 9 8 7 8 7 7
0 0 0 0 0 0 0 0 10 12 9 7 13 7 9
0 0 0 0 0 0 0 0 1238 1230 1236 1235 1230 1240 1234
0 0 0 0 0 0 0 0 … 462 469 467 471 471 468 473)
But, instead of 6x1080, I'd like 256 slots in the histogram to show total number of times each value has appeared. I tried:
julia> hist(img_gld_gs,256)
But that gives:
(2.0:1.0:252.0,
250x1080 Array{Int64,2}:
So instead of a 256x1080 Array, it's 250x1080. Is there any way to force it to have 256 bins (without resorting to writing my own hist function)? I want to be able to compare different images and I want the histogram for each image to have the same number of bins.
Assuming you want a histogram for the entire image (rather than one per row), you might want
hist(vec(img_gld_gs), -1:255)
which first converts the image to a 1-dimensional vector. (You can also use img_gld_gs[:], but that copies the data.)
Also note the range here: the hist function uses a left-open interval, so it will omit counting zeros unless you use something smaller than 0.
hist also accepts a vector (or range) as an optional argument that specifies the edge boundaries, so
hist(img_gld_gs, 0:256)
should work.
I'm getting started with Puppet. I added some lines to a manifest and when running Puppet now it's been at 100% cpu for a very longtime. Is there a good way to see what puppet is actually doing? puppettop?
top gives me this, which is quite useless:
top - 20:02:11 up 1 day, 2:30, 5 users, load average: 1.02, 1.12, 0.93
Tasks: 164 total, 2 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.5%us, 0.0%sy, 0.0%ni, 87.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32809072k total, 10412396k used, 22396676k free, 243832k buffers
Swap: 16768892k total, 0k used, 16768892k free, 6978500k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9761 root 20 0 646m 558m 3568 R 100 1.7 20:42.88 puppet
10509 guaka 20 0 17344 1352 972 R 0 0.0 0:00.28 top
1 root 20 0 24216 2192 1344 S 0 0.0 0:00.85 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:02.83 ksoftirqd/0
4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H