I am trying to adjust a build job within jenkins, the problem is, it keeps failing due to lack of memory, I've adjusted java xmx but it did not solve the problem.
Turns out, I have RAM memory limit within the worker, I tried running those commands as part of the build script : "free -m" and "cat /proc/meminfo" and they both confirmed that job is being run with 1GB RAM limit, the server has more but the build isn't using it and it keeps failing due to lack of memory.
Please help me fix this, how can I lift that limit ? thank you
Heap or Permgen?
There are two OutOfMemoryErrors which people usually encounter. The first is related to heap space: java.lang.OutOfMemoryError: Heap space When you see this, you need to increase the maximum heap space. You can do this by adding the following to your JVM arguments -Xmx200m where you replace the number 200 with the new heap size in megabytes.
The second is related to PermGen: java.lang.OutOfMemoryError: PermGen space. When you see this, you need to increase the maximum Permanent Generation space, which is used for things like class files and interned strings. You can do this by adding the following to your JVM arguments -XX:MaxPermSize=128m where you replace the number 128 with the new PermGen size in megabytes.
Also note:
Memory Requirements for the Master
The amount of memory Jenkins needs is largely dependent on many factors, which is why the RAM allotted for it can range from 200 MB for a small installation to 70+ GB for a single and massive Jenkins master. However, you should be able to estimate the RAM required based on your project build needs.
Each build node connection will take 2-3 threads, which equals about 2 MB or more of memory. You will also need to factor in CPU overhead for Jenkins if there are a lot of users who will be accessing the Jenkins user interface.
It is generally a bad practice to allocate executors on a master, as builds can quickly overload a master’s CPU/memory/etc and crash the instance, causing unnecessary downtime. Instead, it is advisable to set up agents that the Jenkins master can delegate jobs to, keeping the bulk of the work off of the master itself.
Finally, there is a monitoring plugin from Jenkins that you can use:
https://wiki.jenkins.io/display/JENKINS/Monitoring
Sources:
https://wiki.jenkins.io/display/JENKINS/Monitoring
https://wiki.jenkins.io/display/JENKINS/Builds+failing+with+OutOfMemoryErrors
https://www.jenkins.io/doc/book/hardware-recommendations/#:~:text=The%20amount%20of%20memory%20Jenkins,single%20and%20massive%20Jenkins%20master.
Related
I have two different pipeline steps each requiring a different amount of memory. One needs 4096 (size: 2x) memory which has been defined but the other step needs significantly less memory. Is there any way to define the memory for each of the steps in order to not waste pipeline minutes?
Size can be configured for every step.
https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/
We have a server running
Sidekiq 4.2.9
rails 4.2.8
MRI 2.1.9
This server periodically produce some amount of importing from external API's, perform some calculations on them and save these values to the database.
About 3 weeks ago server started hanging, as I see from NewRelic (and when ssh'ed to it) - it consumes more and more memory over time, eventually occupying all available RAM, then server hangs.
I've read some articles about how ruby GC works, but still can't understand, why at ~5:30 AM heap size jumps from ~2.3M to 3M , when there's still 1M free heap slots available(GC settings are default)
similar behavior, 3:35PM:
So, the questions are:
how to make Ruby fill free heap slots instead of requesting new slots from OS ?
how to make it release free heap slots to the system ?
how to make Ruby fill free heap slots instead of requesting new slots from OS ?
Your graph does not have "full" fidelity. It is a lot to assume that GC.stat was called by Newrelic or whatnot just at the exact right time.
It is incredibly likely that you ran out of slots, heap grew and since heaps don't shrink in Ruby you are stuck with a somewhat bloated heap.
To alleviate some of the pain you can limit RUBY_GC_HEAP_GROWTH_MAX_SLOTS to a sane number, something like 100,000 will do, I am trying to lobby setting a default here in core.
Also
Create a persistent log of jobs that run and time they ran (duration and so on), gather GC.stat before and after job runs
Split up your jobs by queue, run 1 queue on one server and other queue on another one, see which queue and which job is responsible for the problem
Profile various jobs you have using flamegraph or other profiling tools
Reduce the amount of concurrent jobs you run as an experiment, or place a mutex between certain job types. It is possible that 1 "job a" at a time is OKish, and 20 concurrent "job a"s at a time will bloat memory.
My Jenkins server is so slow. Java takes 120% or CPU. How can I give Jenkins more memory access.
Of what steps can I take to improve the load time for Jenkins
If Java memory is causing the problem, then you can add more heap via the -Xmx option, as suggested in Priyam's answer. By default, JVM limits heap to 25% of your available RAM.
More heap has a caveat, though: if you add heap in the range of several GB, then the default JVM garbage collection algorithm will periodically impose stop-the-world breaks in the range of several seconds. You then need to switch to a custom garbage collection algorithm (like, CMS) and then carefully tune its parameters.
If adding more heap does not fix your problem, then you need to dig deeper. There's a plethora of possible root causes for a slow master -- from JVM memory and garbage collection settings to plugin issues, on top of the usual CPU/disk/IO-dimensioning issues.
You can allocate more memory and swap heap sizes using the following commands.
These can be set in the job configuration or Jenkins -> Manage -> Configure
-Xmx512m
-XX:MaxPermSize=128m
Manage Jenkins -> Configure System - > Robot Framework
Deselect checkbox as -> [Display "Robot Results" column in the job list view ]
Please look here for detailed screenshot
For demo purposes, I am running Neo4j in a low memory environment -- A laptop with 4GB of RAM, 1644MB is use for video memory, leaving only 2452 MB available for use.. It's also running SQL Server, our WCF services, and our clients.. So there's little memory for Neo4j.
I'm running LOAD CSV cypher scripts via REST from a C# service. There are more than 20 scripts, and theyt work well in a server environment. I've written code to paginate, so that they run in smaller batches. I've reduced the batch size very low ( 25 csv rows ) and a given script may do 300 batches, but I continue to get "Java heap space" errors at some point.
I've tried configuring Neo4j with a relatively large heap space ( 640MB ) which is all the available RAM size plus setting the cache_type to none, and it gets much further before I get the java heap space error. What I don't understand is in that case, why does it grow that much? Also until I restart the neo4j service, I get these java heap space errors quickly. The batch size doesn't seem to impact how much memory is used appreciably.
However, after doing that, and I run the application with these settings, the query performance becomes very slow due to the cache settings.
I am running this on a Windows 7 laptop with 4G RAM -- using Neo4j 2.2.1 Community Edition.
Thoughts?
Perhaps you can share your LOAD CSV statement and the other queries you run.
I think you just run into this:
http://markhneedham.com/blog/2014/10/23/neo4j-cypher-avoiding-the-eager/
So PROFILE or EXPLAIN your queries and make it not to use that much intermediate state. We can help if you share your statements.
And you should use PERIODIC COMMIT 100.
Something like:
heap=512M
dbms.pagecache.memory=200M
keep_logical_logs=false
cache_type=none
http://console.neo4j.org runs neo4j in memory putting up to 50 instances in a single gigabyte of memory. So it should be doable.
In Jenkins I have 100 java projects. Each has its own build file.
Every time I want clear the build file and compile all source files again.
Using bulkbuilder plugin I tried compling all the jobs.. Having 100 jobs run parallel.
But performance is very bad. Individually if the job takes 1 min. in the batch it takes 20mins. More the batch size more the time it takes.. I am running this on powerful server so no problem of memory and CPU.
Please Suggest me how to over come this.. what configurations need to be done in jenkins.
I am launching jenkins using war file.
Thanks..
Even though you say you have enough memory and CPU resources, you seem to imply there is some kind of bottleneck when you increase the number of parallel running jobs. I think this is understandable. Even though I am not a java developer, I think most of the java build tools are able to parallelize build internally. I.e. building a single job may well consume more than one CPU core and quite a lot of memory.
Because of this I suggest you need to monitor your build server and experiment with different batch sizes to find an optimal number. You should execute e.g. "vmstat 5" while builds are running and see if you have idle cpu left. Also keep an eye on the disk I/O. If you increase the batch size but disk I/O does not increase, you are consuming all of the I/O capacity and it probably will not help much if you increase the batch size.
When you have found the optimal batch size (i.e. how many executors to configure for the build server), you can maybe tweak other things to make things faster:
Try to spend as little time checking out code as possible. Instead of deleting workspace before build starts, configure the SCM plugin to remove files that are not under version control. If you use git, you can use a local reference repo or do a shallow clone or something like that.
You can also try to speed things up by using SSD disks
You can get more servers, run Jenkins slaves on them and utilize the cpu and I/O capacity of multiple servers instead of only one.