dockerized cassandra: java.lang.OutOfMemoryError: Java heap space - docker

I have cassandra docker container, which did not start after hard restart pc.
The command line output is very long. This are some snippets
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:26 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
Caused by: java.io.IOException: Corrupt (negative) value length encountered
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
java.lang.OutOfMemoryError: Java heap space
ERROR 20:08:29 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
java.lang.OutOfMemoryError: Java heap space
WARN 20:08:29 ConcurrentMarkSweep GC in 5961ms. CMS Old Gen: 885922704 -> 895736336; Par Survivor Space: 18224024 -> 0
ERROR 20:08:26 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
Thats are some of the error I could pick out. But I do not know how to solve theme. Because of error cointainer do not start, so I could not touch cassandra directly.
One error is repead every time:
ERROR 20:08:26 Unknown exception caught while attempting to update MaterializedView! findkita.kitas

Related

Jenkins 2.304: Failed to monitor for Free Swap Space

Jenkins 2.304, Java 11.
When launching a job to a remote machine, observed a Free Swap Space failure. What could cause it?
Remoting version: 4.10
This is a Unix agent
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Evacuated stdout
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: ERROR: Failed to monitor for Clock Difference
Failed to monitor for Free Swap Space
java.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:65)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:113)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:77)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)

My neo4j server is automatically stopping and starting,

I'm running my neo4j community edition 3.5.5 version with 8GB ram in aws instance.
Initially for few months it ran very fine and got results in millis of time, but now a days it's getting stopping automatically and starting automatically. Sometimes it's not at all starting for hours,even we started it manually also.
Can anyone please help me with this. I'm getting the below logs.
tail -100f /var/log/neo4j/neo4j.log
2019-07-29 13:17:52.570+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 05:33:52.328+0000 WARN The client is unauthorized due to authentication failure.
2019-10-17 15:18:14.652+0000 INFO Transaction with id 2683388 has been automatically rolled back due to transaction timeout.
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid8965.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9050.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)
2019-10-17 17:14:44.651+0000 INFO Transaction with id 2689294 has been automatically rolled back due to transaction timeout.
this can be because you are running lot of merge operations and dont have proper indices created or try increasing the heap size in config file .

WFLYCTL0085: Failed to parse configuration

I am new to Jboss world and running JBOSS and getting following error while starting.
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /home/manish/Desktop/ometap/wildfly-10.0.0.Final
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n
=========================================================================
Listening for transport dt_socket at address: 8787
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by __redirected.__SAXParserFactory (file:/home/manish/Desktop/ometap/wildfly-10.0.0.Final/jboss-modules.jar) to constructor com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl()
WARNING: Please consider reporting this to the maintainers of __redirected.__SAXParserFactory
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
13:34:35,267 INFO [org.jboss.modules] (main) JBoss Modules version 1.5.1.Final
13:34:35,627 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
13:34:35,798 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) starting
13:34:36,608 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0055: Caught exception during boot: org.jboss.as.controller.persistence.ConfigurationPersistenceException: WFLYCTL0085: Failed to parse configuration
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:131)
at org.jboss.as.server.ServerService.boot(ServerService.java:356)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:299)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.xml.stream.XMLStreamException: WFLYCTL0083: Failed to load module org.wildfly.extension.core-management
at org.jboss.as.controller.parsing.ExtensionXml.parseExtensions(ExtensionXml.java:155)
at org.jboss.as.server.parsing.StandaloneXml$DefaultExtensionHandler.parseExtensions(StandaloneXml.java:126)
at org.jboss.as.server.parsing.StandaloneXml_4.readServerElement(StandaloneXml_4.java:218)
at org.jboss.as.server.parsing.StandaloneXml_4.readElement(StandaloneXml_4.java:141)
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:103)
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:49)
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110)
at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69)
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:123)
... 3 more
Caused by: java.util.concurrent.ExecutionException: javax.xml.stream.XMLStreamException: WFLYCTL0083: Failed to load module
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.jboss.as.controller.parsing.ExtensionXml.parseExtensions(ExtensionXml.java:147)
... 11 more
Caused by: javax.xml.stream.XMLStreamException: WFLYCTL0083: Failed to load module
at org.jboss.as.controller.parsing.ExtensionXml.loadModule(ExtensionXml.java:196)
at org.jboss.as.controller.parsing.ExtensionXml.access$000(ExtensionXml.java:69)
at org.jboss.as.controller.parsing.ExtensionXml$1.call(ExtensionXml.java:127)
at org.jboss.as.controller.parsing.ExtensionXml$1.call(ExtensionXml.java:124)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.jboss.modules.ModuleNotFoundException: org.wildfly.extension.core-management:main
at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:223)
at org.jboss.as.controller.parsing.ExtensionXml.loadModule(ExtensionXml.java:178)
... 8 more
13:34:36,611 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
13:34:36,615 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested.
13:34:36,681 INFO [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) stopped in 34ms
How can i fix above error?
If you want to use Java 9+ I'd suggest using at least WildFly 15. WildFly 17.0.1.Final is the latest release and does work with Java 9+.
If you are stuck on WildFly 10, I'd suggest using Java 1.8 instead.

Jenkins gives error "Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"?

While Building a project in jenkins I am getting an Error of OutofBoundMemory.
the log is like this..
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.sonar.sslr.internal.vm.Machine.createNode(Machine.java:256)
at org.sonar.sslr.internal.vm.Instruction$RetInstruction.execute(Instruction.java:305)
at org.sonar.sslr.internal.vm.Machine.execute(Machine.java:162)
at org.sonar.sslr.internal.vm.Machine.execute(Machine.java:106)
at org.sonar.sslr.internal.vm.Machine.parse(Machine.java:84)
at org.sonar.sslr.parser.ParseRunner.parse(ParseRunner.java:45)
at com.sonar.sslr.api.typed.ActionParser.parse(ActionParser.java:102)
at com.sonar.sslr.api.typed.ActionParser.parse(ActionParser.java:91)
at org.sonar.php.PHPAnalyzer.nextFile(PHPAnalyzer.java:71)
at org.sonar.plugins.php.PHPSensor.analyseFile(PHPSensor.java:142)
at org.sonar.plugins.php.PHPSensor.analyseFiles(PHPSensor.java:124)
at org.sonar.plugins.php.PHPSensor.analyse(PHPSensor.java:115)
at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:58)
at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:50)
at org.sonar.batch.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:83)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:192)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:241)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:226)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.task.ScanTask.execute(ScanTask.java:47)
at org.sonar.batch.task.TaskContainer.doAfterStart(TaskContainer.java:86)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)
at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:106)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:119)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Even the build continues for I guess long time.. I am using sonarqube runner
What should I do?
In the build step's "JVM Options" input, you need to specify more memory (than the default) for the process. Here's what that might look like:
-Xmx2g -Xms512m -XX:MaxPermSize=512m
Note that you should adjust these values based on your system resources.

Grails PermGem error

I need help with this problem. I need someone to explain to me why is this happening and how to prevent or avoid it.
Exception in thread "Thread-747" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-748" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-759" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-760" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-764" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-765" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-766" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-767" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-773" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-774" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-780" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-781" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-788" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Thread-789" java.lang.OutOfMemoryError: PermGen space
2011-06-20 14:42:10,668 [http-8080-6] ERROR [/CM].[grails] - Servlet.service() for servlet grails threw exception
java.lang.OutOfMemoryError: PermGen space
2011-06-20 14:42:10,668 [http-8080-6] ERROR [/CM].[default] - Servlet.service() for servlet default threw exception
java.lang.OutOfMemoryError: PermGen space
: java.lang.OutOfMemoryError: PermGen space
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:116)
at _GrailsPackage_groovy$_run_closure8.doCall(_GrailsPackage_groovy:275)
at _GrailsPackage_groovy$_run_closure8.call(_GrailsPackage_groovy)
at _GrailsRun_groovy$_run_closure8.doCall(_GrailsRun_groovy:245)
at RunApp$_run_closure1.doCall(RunApp.groovy:35)
at gant.Gant$_dispatch_closure5.doCall(Gant.groovy:381)
at gant.Gant$_dispatch_closure7.doCall(Gant.groovy:415)
at gant.Gant$_dispatch_closure7.doCall(Gant.groovy)
at gant.Gant.withBuildListeners(Gant.groovy:427)
at gant.Gant.this$2$withBuildListeners(Gant.groovy)
at gant.Gant$this$2$withBuildListeners.callCurrent(Unknown Source)
at gant.Gant.dispatch(Gant.groovy:415)
at gant.Gant.this$2$dispatch(Gant.groovy)
at gant.Gant.invokeMethod(Gant.groovy)
at gant.Gant.executeTargets(Gant.groovy:590)
at gant.Gant.executeTargets(Gant.groovy:589)
Caused by: java.lang.OutOfMemoryError: PermGen space
--- Nested Exception ---
java.lang.OutOfMemoryError: PermGen space
Error automatically restarting container: java.lang.OutOfMemoryError: PermGen space
Error executing script RunApp: PermGen space
java.lang.OutOfMemoryError: PermGen space
Error executing script RunApp: PermGen space
Application context shutting down...
Application context shutdown.
The PermGen is a region of your JVM's memory that is used to load classes.
As you application executes, it uses more and more of this memory, especially if you are in a debugging environment, of if you make extensive use of closures.
The way to fix this is to add more of it!
This is done by passing one or two parameters to the JVM when launching your application.
The parameters are :
-XX:MaxPermSize=256m
-XX:PermSize=128m
(adjust the values to your specific needs)
The PermSize will be the initial size of the PermGen, and the MaxPermSize will be the maximum size at which it will increase before throwing you an exception like in your post.
By default, it is set to 64M, which is not much if you have a 'real' application.
PAY ATTENTION : Your total memory usage will be:Heap size + Perm Size
If you are using Servlet Version 3.0 then even increasing your memory won't be of any help since it's a problem with the groovy compiler. The new version 1.8.2/1.9 (?) which will be released soon will resolve this issue. In the meantime you can change the servlet version back to "2.5" (in BuildConfig.groovy) which will resolve this issue.
The drawback of changing the servlet version to 2.5 is that it cannot be deployed to Glassfish application server so the ugly workaround is change to 2.5 and use "run-app". When you want to deploy to glassfish change the servlet version to "3.0" in BuildConfig.groovy, run "war" and then deploy the war to Glassfish.
Change it back to "2.5" to run on local dev machine again.
Check the FAQ
Q: OMG I get OutOfMemoryErrors or PermGen Space errors when running Grails in development mode. What do I do?
Since Grails 0.6, Grails automatically re-compiles Java sources and
domain classes using pre-compilation and then a server restart. This
can lead to permgen space running out if the server is run for a long
time and many changes are made. You can disable this feature if it is
not important to you with:
grails -Ddisable.auto.recompile=true run-app
There is also a problem with Grails 0.6 on Windows where you get
OutOfMemoryErrors during a period of activity in development mode due
to the re-compilation. This may be solved in SVN head, but if you see
this problem the above option can also help.
Easiest just to restart your application server when it happens.
In the STS IDE set as per following:
-XX:MaxPermSize=512m -XX:PermSize=128m
I hope it helps.

Resources