Jenkins getting stopped automatically in linux server - jenkins

My jenkins server is getting stopped while the jenkins pipeline is running.
I don't know why its happening
Can anyone help me on this
Thanks in advance.
Jenkins logs
2022-02-28 11:36:16.019+0000 [id=78] INFO o.j.p.workflow.job.WorkflowRun#finish: pipeline #82 completed: SUCCESS
2022-02-28 12:33:52.148+0000 [id=230] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 12:33:52.171+0000 [id=230] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 14 ms
2022-02-28 13:04:29.255+0000 [id=278] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started telemetry collection
2022-02-28 13:04:30.277+0000 [id=278] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished telemetry collection. 1,015 ms
2022-02-28 13:33:52.148+0000 [id=332] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 13:33:52.158+0000 [id=332] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-02-28 14:33:52.148+0000 [id=425] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 14:33:52.158+0000 [id=425] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-02-28 15:33:52.148+0000 [id=526] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 15:33:52.156+0000 [id=526] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 4 ms
2022-02-28 16:33:52.149+0000 [id=631] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 16:33:52.161+0000 [id=631] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 7 ms
2022-02-28 16:36:45.550+0000 [id=646] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Fingerprint cleanup
2022-02-28 16:36:45.660+0000 [id=646] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Fingerprint cleanup. 106 ms
2022-02-28 17:17:51.445+0000 [id=704] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Workspace clean-up
2022-02-28 17:17:51.570+0000 [id=704] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Workspace clean-up. 121 ms
2022-02-28 17:33:52.149+0000 [id=732] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 17:33:52.157+0000 [id=732] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 4 ms
2022-02-28 18:33:52.149+0000 [id=827] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 18:33:52.157+0000 [id=827] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-02-28 19:33:52.148+0000 [id=914] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 19:33:52.162+0000 [id=914] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-02-28 20:33:52.149+0000 [id=1013] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 20:33:52.156+0000 [id=1013] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 4 ms
2022-02-28 21:33:52.149+0000 [id=1102] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 21:33:52.180+0000 [id=1102] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 28 ms
2022-02-28 22:33:52.149+0000 [id=1188] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 22:33:52.161+0000 [id=1188] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 9 ms
2022-02-28 23:33:52.148+0000 [id=1289] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-02-28 23:33:52.156+0000 [id=1289] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-03-01 00:33:52.148+0000 [id=1376] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-03-01 00:33:52.157+0000 [id=1376] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 5 ms
2022-03-01 01:33:52.148+0000 [id=1469] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
2022-03-01 01:33:52.156+0000 [id=1469] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 6 ms
2022-03-01 02:33:52.148+0000 [id=1554] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder

Related

In EKS, Worker pods going offline abruptly with 'hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel JNLP4-connect connection'

Our Environment:
Jenkins version - Jenkins 2.319.1
Jenkins Master image : jenkins/jenkins:2.319.1-lts-alpine
Jenkins worker image: jenkins/inbound-agent:4.11-1-alpine
Installed plugins:
Kubernetes - 1.30.6
Kubernetes Client API - 5.4.1
Kubernetes Credentials Plugin - 0.9.0
JAVA version on master: openjdk 11.0.13
JAVA version on Agent/worker : openjdk 11.0.14
Hi team,
We are facing issue in jenkins where jenkins agent disconnects(or goes offline) from master while still job is running on agent/worker. We are getting below error(highlighted) and tried below things but issue is still not resolving fully. Jenkins is deployed on EKS.
Error:
5334535:2022-11-02 14:07:54.573+0000 [id=140290] INFO hudson.slaves.NodeProvisioner#update: worker-7j4x4 provisioning successfully completed. We have now 2 computer(s)
5334695:2022-11-02 14:07:54.675+0000 [id=140291] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes done-jenkins/worker-7j4x4
5334828:2022-11-02 14:07:56.619+0000 [id=140291] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes done-jenkins/worker-7j4x4
5334964-2022-11-02 14:07:58.650+0000 [id=140309] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #97 from /100.122.254.111:42648
5335123-2022-11-02 14:09:19.733+0000 [id=140536] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work
5335275-2022-11-02 14:09:19.733+0000 [id=140536] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
5335409-2022-11-02 14:09:19.734+0000 [id=140536] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 2608, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
5335965-2022-11-02 14:09:19.734+0000 [id=140536] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 1 nodes assigned to this Jenkins instance, which we will check
5336139-2022-11-02 14:09:19.734+0000 [id=140536] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
5336279-2022-11-02 14:09:19.734+0000 [id=140536] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
5336438-groovy.lang.MissingPropertyException: No such property: envVar for class: groovy.lang.Binding
5336532- at groovy.lang.Binding.getVariable(Binding.java:63)
5336585- at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:271)
–
5394279-2022-11-02 15:09:19.733+0000 [id=141899] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work
5394431-2022-11-02 15:09:19.734+0000 [id=141899] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
5394565-2022-11-02 15:09:19.734+0000 [id=141899] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 2620, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
5395121-2022-11-02 15:09:19.734+0000 [id=141899] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 3 nodes assigned to this Jenkins instance, which we will check
5395295-2022-11-02 15:09:19.734+0000 [id=141899] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
5395435-2022-11-02 15:09:19.734+0000 [id=141899] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
5395594-2022-11-02 15:11:59.502+0000 [id=140320] INFO hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel JNLP4-connect connection from ip-100-122-254-111.eu-central-1.compute.internal/100.122.254.111:42648.
5395817-java.util.concurrent.TimeoutException: Ping started at 1667401679501 hasn't completed by 1667401919502
5395920- at hudson.remoting.PingThread.ping(PingThread.java:134)
5395977- at hudson.remoting.PingThread.run(PingThread.java:90)
5396032:2022-11-02 15:11:59.503+0000 [id=141914] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting 5049 for worker-7j4x4 terminated: java.nio.channels.ClosedChannelException
5396231-2022-11-02 15:12:35.579+0000 [id=141933] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
5396368-2022-11-02 15:12:36.257+0000 [id=141933] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 678 ms
5396514-2022-11-02 15:14:15.582+0000 [id=141422] INFO hudson.slaves.ChannelPinger$1#onDead: Ping failed. Terminating the channel JNLP4-connect connection from ip-100-122-237-38.eu-central-1.compute.internal/100.122.237.38:55038.
5396735-java.util.concurrent.TimeoutException: Ping started at 1667401815582 hasn't completed by 1667402055582
5396838- at hudson.remoting.PingThread.ping(PingThread.java:134)
5396895- at hudson.remoting.PingThread.run(PingThread.java:90)
5396950-2022-11-02 15:14:15.584+0000 [id=141915] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting 5050 for worker-fjf1p terminated: java.nio.channels.ClosedChannelException
****5397149-2022-11-02 15:14:19.733+0000 [id=141950] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work
5397301-2022-11-02 15:14:19.733+0000 [id=141950] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
5397435-2022-11-02 15:14:19.734+0000 [id=141950] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 2621, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
Any suggestion or resolutions pls.
Tried below things:
Increased idleMinutes to 180 from default
Verified that resources are sufficient as per graphana dashboard
Changed podRetention to onFailure from Never
Changed podRetention to Always from Never
Increased readTimeout
Increased connectTimeout
Increased slaveConnectTimeoutStr
Disabled the ping thread from UI via disabling “response time" checkbox from preventive node monitroing
Increased activeDeadlineSeconds
Verified same java version on master and agent
Updated kubernetes and kubernetes API client plugins
Expectation is worker/agent should disconnect once job is successfully ran and after idleMinutes defined it should terminate but few times its terminating while job is still running on agent

Jenkins keep cleaning up my workspace unexpectedly

I have Jenkins in my Docker container to run the testing.
I noticed a weird thing that my workspace was removed every day.
Check the docker logs:
2021-11-26 08:26:57.511+0000 [id=450812] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 7,676 ms
2021-11-27 08:26:49.834+0000 [id=461439] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-11-27 08:26:49.835+0000 [id=461439] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 0 ms
2021-11-28 08:26:49.834+0000 [id=479487] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-11-28 08:26:49.835+0000 [id=479487] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 0 ms
2021-11-29 08:26:49.834+0000 [id=497481] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-11-29 08:26:49.835+0000 [id=497481] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 0 ms
2021-11-30 08:26:49.834+0000 [id=515511] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-11-30 08:26:52.726+0000 [id=515511] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 2,891 ms
2021-12-01 08:26:49.834+0000 [id=516550] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-12-01 08:27:00.562+0000 [id=516550] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 10,727 ms
2021-12-02 08:26:49.834+0000 [id=520493] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-12-02 08:26:59.681+0000 [id=520493] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 9,847 ms
Looks like Jenkins removes the workspace every 24 hours.
The weird thing is I do have a workspace cleanup plugin in my Jenkins but I have disabled it in the Plugin Manger and I don't have any workspace cleanup plugin syntax in my pipeline script.
Is there any other default feature/function/plug-in that will remove my workspace periodically?
I am very new to Jenkins, any help is appreciated!
######################## Updated on 12/4 ########################
I found this discussion is very similar to my question:
Jenkins is deleting workspaces on slaves
I changed this parameter to true in the Script Console.
hudson.model.WorkspaceCleanupThread.disabled = true
The weird thing is that the workspace clean up still happen, but at least this time it didn't remove the folder:
2021-12-03 21:15:02.446+0000 [id=19935] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-12-03 21:15:05.225+0000 [id=19935] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 2,778 ms
2021-12-04 21:15:02.447+0000 [id=38002] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Workspace clean-up
2021-12-04 21:15:02.447+0000 [id=38002] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Workspace clean-up. 0 ms

Failed to start Neo4j after successful upgrade

I installed neo4j 3.0.3. I took 2.3.0-M02 database, enabled upgrade by setting uncommenting
dbms.allow_format_migration=true
and started server, but it fails with
2016-06-15 17:50:18.439+0000 INFO Starting...
2016-06-15 17:50:18.874+0000 INFO Bolt enabled on localhost:7687.
2016-06-15 17:50:19.027+0000 INFO Starting upgrade of database
2016-06-15 17:50:19.046+0000 INFO Migrating Indexes (1/3):
2016-06-15 17:50:19.049+0000 INFO 10% completed
2016-06-15 17:50:19.049+0000 INFO 20% completed
2016-06-15 17:50:19.049+0000 INFO 30% completed
2016-06-15 17:50:19.050+0000 INFO 40% completed
2016-06-15 17:50:19.050+0000 INFO 50% completed
2016-06-15 17:50:19.050+0000 INFO 60% completed
2016-06-15 17:50:19.050+0000 INFO 70% completed
2016-06-15 17:50:19.050+0000 INFO 80% completed
2016-06-15 17:50:19.050+0000 INFO 90% completed
2016-06-15 17:50:19.050+0000 INFO 100% completed
2016-06-15 17:50:19.051+0000 INFO Migrating Legacy indexes (2/3):
2016-06-15 17:50:19.057+0000 INFO 10% completed
2016-06-15 17:50:19.058+0000 INFO 20% completed
2016-06-15 17:50:19.058+0000 INFO 30% completed
2016-06-15 17:50:19.058+0000 INFO 40% completed
2016-06-15 17:50:19.058+0000 INFO 50% completed
2016-06-15 17:50:19.058+0000 INFO 60% completed
2016-06-15 17:50:19.058+0000 INFO 70% completed
2016-06-15 17:50:19.058+0000 INFO 80% completed
2016-06-15 17:50:19.058+0000 INFO 90% completed
2016-06-15 17:50:19.059+0000 INFO 100% completed
2016-06-15 17:50:19.059+0000 INFO Migrating Store files (3/3):
2016-06-15 17:50:19.067+0000 INFO 10% completed
2016-06-15 17:50:19.067+0000 INFO 20% completed
2016-06-15 17:50:19.067+0000 INFO 30% completed
2016-06-15 17:50:19.068+0000 INFO 40% completed
2016-06-15 17:50:19.068+0000 INFO 50% completed
2016-06-15 17:50:19.068+0000 INFO 60% completed
2016-06-15 17:50:19.068+0000 INFO 70% completed
2016-06-15 17:50:19.068+0000 INFO 80% completed
2016-06-15 17:50:19.068+0000 INFO 90% completed
2016-06-15 17:50:19.068+0000 INFO 100% completed
2016-06-15 17:50:19.091+0000 INFO Successfully finished upgrade of database
2016-06-15 17:50:19.389+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3693c13' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3693c13' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3693c13' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:217)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:87)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:66)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3693c13' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:189)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/pavel/devel/neo4j-community-3.0.3/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:40)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer.lambda$static$31(CommunityNeoServer.java:55)
at org.neo4j.server.CommunityNeoServer$$Lambda$43/1644443712.newGraphDatabase(Unknown Source)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine#6ec66b58' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:503)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:99)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
... 11 more
Caused by: org.neo4j.kernel.impl.store.InvalidRecordException: DynamicRecord[1869509480,used=false,(0),type=-1,data=byte[],start=true,next=-1] not in use
at org.neo4j.kernel.impl.store.record.RecordLoad.verify(RecordLoad.java:63)
at org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterNotRead(CommonAbstractStore.java:1229)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1051)
at org.neo4j.kernel.impl.store.CommonAbstractStore.access$000(CommonAbstractStore.java:60)
at org.neo4j.kernel.impl.store.CommonAbstractStore$1.next(CommonAbstractStore.java:1177)
at org.neo4j.kernel.impl.store.CommonAbstractStore$1.next(CommonAbstractStore.java:1142)
at org.neo4j.kernel.impl.store.RecordCursor.getAll(RecordCursor.java:106)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecords(CommonAbstractStore.java:1124)
at org.neo4j.kernel.impl.store.TokenStore.ensureHeavy(TokenStore.java:141)
at org.neo4j.kernel.impl.store.TokenStore.getStringFor(TokenStore.java:146)
at org.neo4j.kernel.impl.store.TokenStore.getTokens(TokenStore.java:100)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.start(RecordStorageEngine.java:415)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 19 more
I tried to google this one up, but there not much to be found, except the code itself.
Upgrades from milestones are usually not supported.
Can you check that the old store was ok with the consistency checker?
http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/
You could also try to use my store-copy tool on your store to copy over all non-affected records: https://github.com/jexp/store-utils/tree/30

Hazelcast memory is continuously increasing

I have a hazelcast cluster with two machines.
The only object in the cluster is a map. Analysing the log files I noticed that the health monitor starts to report a slow increase in memory consumption even though no new entries are being added to map (see sample of log entries below)
Any ideas of what may be causing the memory increase?
<p>2015-09-16 10:45:49 INFO HealthMonitor:? - [10.11.173.129]:5903
[dev] [3.2.1] memory.used=97.6M, memory.free=30.4M,
memory.total=128.0M, memory.max=128.0M, memory.used/total=76.27%,
memory.used/max=76.27%, load.process=0.00%, load.system=1.00%,
load.systemAverage=3.00%, thread.count=96, thread.peakCount=107,
event.q.size=0, executor.q.async.size=0, executor.q.client.size=0,
executor.q.operation.size=0, executor.q.query.size=0,
executor.q.scheduled.size=0, executor.q.io.size=0,
executor.q.system.size=0, executor.q.operation.size=0,
executor.q.priorityOperation.size=0, executor.q.response.size=0,
operations.remote.size=1, operations.running.size=0, proxy.count=2,
clientEndpoint.count=0, connection.active.count=2,
connection.count=2</p>
<p>2015-09-16 10:46:02 INFO
InternalPartitionService:? - [10.11.173.129]:5903 [dev] [3.2.1]
Remaining migration tasks in queue = 51 2015-09-16 10:46:12 DEBUG
TeleavisoIvrLoader:71 - Checking for new files... 2015-09-16 10:46:13
INFO InternalPartitionService:? - [10.11.173.129]:5903 [dev] [3.2.1]
All migration tasks has been completed, queues are empty. 2015-09-16
10:46:19 INFO HealthMonitor:? - [10.11.173.129]:5903 [dev] [3.2.1]
memory.used=103.9M, memory.free=24.1M, memory.total=128.0M,
memory.max=128.0M, memory.used/total=81.21%, memory.used/max=81.21%,
load.process=0.00%, load.system=1.00%, load.systemAverage=2.00%,
thread.count=73, thread.peakCount=107, event.q.size=0,
executor.q.async.size=0, executor.q.client.size=0,
executor.q.operation.size=0, executor.q.query.size=0,
executor.q.scheduled.size=0, executor.q.io.size=0,
executor.q.system.size=0, executor.q.operation.size=0,
executor.q.priorityOperation.size=0, executor.q.response.size=0,
operations.remote.size=0, operations.running.size=0, proxy.count=2,
clientEndpoint.count=0, connection.active.count=2,
connection.count=2</p>
<p>2015-09-16 10:46:49 INFO HealthMonitor:? - [10.11.173.129]:5903
[dev] [3.2.1] memory.used=105.1M, memory.free=22.9M,
memory.total=128.0M, memory.max=128.0M, memory.used/total=82.11%,
memory.used/max=82.11%, load.process=0.00%, load.system=1.00%,
load.systemAverage=1.00%, thread.count=73, thread.peakCount=107,
event.q.size=0, executor.q.async.size=0, executor.q.client.size=0,
executor.q.operation.size=0, executor.q.query.size=0,
executor.q.scheduled.size=0, executor.q.io.size=0,
executor.q.system.size=0, executor.q.operation.size=0,
executor.q.priorityOperation.size=0, executor.q.response.size=0,
operations.remote.size=0, operations.running.size=0, proxy.count=2,
clientEndpoint.count=0, connection.active.count=2,
connection.count=2</p>

Neo4j randomly shutting down

I am running neo4j on an EC2 instance. But for some reason it randomly shuts down from time to time. Is there a way to check the shutdown logs? And is there a way to automatically restart the server? I couldn't locate the log folder. But here's what my messages.log file looks like. This section covers the timeframe when the server went down (before 2015-04-13 05:39:59.084+0000) and when I manually restarted the server (at 2015-04-13 05:39:59.084+0000). You can see that there is no record of server issue or shutdown. Time frame before 2015-03-05 08:18:47.084+0000 contains info of the previous server restart.
2015-03-05 08:18:44.180+0000 INFO [o.n.s.m.Neo4jBrowserModule]: Mounted Neo4j Browser at [/browser]
2015-03-05 08:18:44.253+0000 INFO [o.n.s.w.Jetty9WebServer]: Mounting static content at [/webadmin] from [webadmin-html]
2015-03-05 08:18:44.311+0000 INFO [o.n.s.w.Jetty9WebServer]: Mounting static content at [/browser] from [browser]
2015-03-05 08:18:47.084+0000 INFO [o.n.s.CommunityNeoServer]: Server started on: http://0.0.0.0:7474/
2015-03-05 08:18:47.084+0000 INFO [o.n.s.CommunityNeoServer]: Remote interface ready and available at [http://0.0.0.0:7474/]
2015-03-05 08:18:47.084+0000 INFO [o.n.k.i.DiagnosticsManager]: --- SERVER STARTED END ---
2015-04-13 05:39:59.084+0000 INFO [o.n.s.CommunityNeoServer]: Setting startup timeout to: 120000ms based on -1
2015-04-13 05:39:59.265+0000 INFO [o.n.k.InternalAbstractGraphDatabase]: No locking implementation specified, defaulting to 'community'
2015-04-13 05:39:59.383+0000 INFO [o.n.k.i.DiagnosticsManager]: --- INITIALIZED diagnostics START ---
2015-04-13 05:39:59.384+0000 INFO [o.n.k.i.DiagnosticsManager]: Neo4j Kernel properties:
2015-04-13 05:39:59.389+0000 INFO [o.n.k.i.DiagnosticsManager]: neostore.propertystore.db.mapped_memory=78M
2015-04-13 05:39:59.389+0000 INFO [o.n.k.i.DiagnosticsManager]: neostore.nodestore.db.mapped_memory=21M

Resources