I set up the release plugin on my Grails project and successfully ran it on my localhost.
When I try to set up the same build in Jenkins, the build hangs indefinitely. The last thing in the output before it hangs is the checkCommitNeeded step.
Anything I can do to figure out what's going wrong?
I have set -Prelease.useAutomaticVersion=true and the two version params in switches, as mentioned in the plugin docs.
Update
On the researchgate Gitter, Christian Gonzalez mentioned that Jenkins is detecting another commit caused by the release plugin, and getting itself stuck in a loop. For Git, an additional behavior can be added to ignore changes committed by the plugin. However, my project is using SVN.
Update
Below is a snippet of the output from adding -d
11:12:48.907 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter] Executing actions for task ':checkCommitNeeded'.
11:12:48.908 [INFO] [org.gradle.api.Project] Running [svn, status] in [/var/lib/jenkins/jobs/MyTeam/jobs/MyProject/jobs/MyProject-release/workspace]
11:12:48.924 [INFO] [org.gradle.api.Project] Running [svn, status] produced output: []
11:12:48.926 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':checkCommitNeeded'
11:12:48.926 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :checkCommitNeeded (Thread[Daemon worker,5,main]) completed. Took 0.02 secs.
11:12:48.926 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationWorkerRegistry] Worker root.3 completed (0 in use)
11:12:48.926 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationWorkerRegistry] Worker root.4 started (1 in use).
11:12:48.926 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :checkUpdateNeeded (Thread[Daemon worker,5,main]) started.
11:12:48.927 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] :myproject:checkUpdateNeeded
11:12:48.927 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Starting to execute task ':checkUpdateNeeded'
11:12:48.927 [DEBUG] [org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter] Determining if task ':checkUpdateNeeded' is up-to-date
11:12:48.927 [INFO] [org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter] Executing task ':checkUpdateNeeded' (up-to-date check took 0.0 secs) due to:
Task has not declared any outputs.
11:12:48.927 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter] Executing actions for task ':checkUpdateNeeded'.
11:12:48.928 [INFO] [org.gradle.api.Project] Running [svn, status, -q, -u] in [/var/lib/jenkins/jobs/MyTeam/jobs/MyProject/jobs/MyProject-release/workspace]
11:12:51.477 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] DaemonExpirationPeriodicCheck running
11:12:51.479 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
11:12:51.480 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired.
11:12:51.481 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
11:13:01.477 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] DaemonExpirationPeriodicCheck running
11:13:01.477 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
11:13:01.478 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired.
11:13:01.480 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
11:13:11.477 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] DaemonExpirationPeriodicCheck running
11:13:11.477 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
11:13:11.477 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired.
11:13:11.479 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
...
The last 4 lines are repeated over and over.
I faced the same issue. For me, the reason was that I did the wrong setup configuration for the project. Example: wrong GitHub URL (without .git extension) added, incorrect Poll SCM config, etc.
Fix for me was to restart the Jenkins server & correct the changes under 'Manage' for your project & again build.
Related
I am running an AngularDart project in a Docker Container google/dart and install webdev.
It is running very slow, so I have tried to add more cpu - do you know how to optimise and make it faster and is it good practice to run AngularDart in google/dart?
docker container run --name webdev -it --rm -p 8080:8080 -v $(pwd):/usr/src/app --cpu-shares=2 google/dart
The first time, it builds ok, but after making changes - the second build is very slow.
You can see the build log here:
webdev serve --hostname 0.0.0.0
Creating build script, took 22350ms
[INFO] Setting up file watchers completed, took 135ms
[INFO] Waiting for all file watchers to be ready completed, took 5.6s
[INFO] Reading cached asset graph completed, took 2.9s
[INFO] Checking for updates since last build completed, took 4.3s
[INFO] Running build completed, took 1.9s
[INFO] Caching finalized dependency graph completed, took 1.5s
[INFO] Succeeded after 3.5s with 0 outputs (0 actions)
Serving `web` on http://0.0.0.0:8080
Serving `test` on http://0.0.0.0:8081
[INFO] ------------------------------------------------------------------------
[INFO] Starting Build
[INFO] Updating asset graph completed, took 57ms
[WARNING] No actions completed for 32.6s, waiting on:
- angular on lib/app_component.dart
[WARNING] No actions completed for 15.0s, waiting on:
- build_web_compilers|ddc on lib/app_component.template.dartdevc.module
[WARNING] No actions completed for 15.1s, waiting on:
- build_web_compilers|ddc on lib/app_component.template.dartdevc.module
[WARNING] No actions completed for 20.6s, waiting on:
- build_modules|dartdevc on lib/app_component.template.dartdevc.module
[WARNING] No actions completed for 31.2s, waiting on:
- build_modules|dartdevc on lib/app_component.template.dartdevc.module
[INFO] Running build completed, took 2m 28s
[INFO] Caching finalized dependency graph completed, took 3.1s
[INFO] Succeeded after 2m 31s with 12 outputs (10 actions)
I'm using sos berlin scheduler (version linux-x64 1.10.5).
Normally when job in job_chain timeout, scheduler will kill the job process and send a email.
So, based on this, I want to trigger other job.
But, I have tried two ways all doesn't work.
Way 1:
Add a function “spooler_task_after()” in the job.
I guess the failure is because this job will create a process on linux system, while job timeout scheduler kill this job process, also kill the function “spooler_task_after()”
Code:
<job timeout="00:00:09">
<script language="shell"><![CDATA[
echo aa
sleep 10s
echo bb
]]></script>
<monitor name="exit_code" ordering="0">
<script language="java:javascript"><![CDATA[
function spooler_task_after(){
var exitCode = spooler_task.exit_code;
spooler_log.info ("Exit Code is: " + exitCode);
/*
call other job
*/
result = true;
return result;
}
]]></script>
</monitor>
<run_time/>
</job>
Result:
2017-07-27 21:22:21.251+0800 [info]
2017-07-27 21:22:21.251+0800 [info] Task sample_errorhandling/job1:23026 - Protocol starts in /httx/opt/sos-scheduler/ldw-scheduler-test1/logs/task.sample_errorhandling,job1.log
2017-07-27 21:22:21.250+0800 [info] SCHEDULER-842 Task is going to process Order sample_errorhandling/job_chain3:12, state=aaa, on JobScheduler 'http://xxxx:4444', Order's Process_class
2017-07-27 21:22:21.268+0800 [info] SCHEDULER-726 Task runs on this JobScheduler 'http://jt-host-kvm-72:4444'
2017-07-27 21:22:21.268+0800 [info] SCHEDULER-918 state=starting (at=never)
2017-07-27 21:22:22.466+0800 [info] SCHEDULER-987 Starting process: '/bin/sh' '-c' '"/tmp/admin/sos.gBdCm8"'
2017-07-27 21:22:23.520+0800 [info] [stdout] aa
2017-07-27 21:22:30.326+0800 [ERROR] SCHEDULER-272 Terminating task after reaching deadline <job timeout="9">
2017-07-27 21:22:30.359+0800 [ERROR] SCHEDULER-202 Connection to task has been lost, state=running_remote_process: Z-REMOTE-101 Separate process: pid=0: Connection lost / zschimmer::com::object_server::Connection::pop_operation
2017-07-27 21:22:30.359+0800 [ERROR] SCHEDULER-202 Connection to task has been lost, state=release: Z-REMOTE-122 Separate process pid=0: Caller has killed process
2017-07-27 21:22:30.384+0800 [ERROR] SCHEDULER-280 Process terminated with exit code 1 (0x63)
2017-07-27 21:22:30.384+0800 [WARN] SCHEDULER-845 Task ended without processing the order. The order remains in job's order queue in the same state
2017-07-27 21:22:30.384+0800 [info] SCHEDULER-843 Task has ended processing of Order sample_errorhandling/job_chain3:12, state=aaa, on JobScheduler 'http:/xxxx:4444'
Way 2:
Add return code on job chain node
This way works on job execute successfully or with error. But failed when job was killed with timeout.
Code in job chain:
<job_chain >
<job_chain_node state="aaa" job="job1" next_state="success" error_state="error">
<on_return_codes >
<on_return_code return_code="1">
<add_order xmlns="https://jobscheduler-plugins.sos-berlin.com/NodeOrderPlugin" job_chain="/error_handling/sendmail"/>
</on_return_code>
</on_return_codes>
</job_chain_node>
<job_chain_node state="success"/>
<job_chain_node state="error"/>
</job_chain>
You can use the error_state= attribute.
When JobScheduler kills the task because of a timeout this is handled as an error situation.
Please note the the next_state of the errorHandling state is error to indicate in JOC that this was an error and that the errorHandling state have its own error_state to indicate if the errorHandler itself fails.
<job_chain>
<job_chain_node state="100" job="job1" next_state="200" error_state="errorHandling"/>
<job_chain_node state="200" job="job2" next_state="success" error_state="errorHandling"/>
<job_chain_node state="errorHandling" job="errorHandlerJob" next_state="error" error_state="errorInErrorHandling"/>
<job_chain_node state="success"/>
<job_chain_node state="errorInErrorHandling"/>
<job_chain_node state="error"/>
</job_chain>
I have a project with hundreds java modules.
I run SonarQube 5.1 with following plugins:
Java 3.3
Findbugs 3.2
Checkstyle 2.3
PMD 2.4.1
Issue Assign 1.6
SQALE 2.6
SVN 1.1
LDAP 1.4
JIRA 1.2
Cobertura 1.6.3
I use jenkins 1.639 with SonarQube plugin 2.3.
I set a job with the following settings:
Goals :
$SONAR_MAVEN_GOAL -Dsonar.host.url=$SONAR_HOST_URL -Dsonar.jdbc.url=$SONAR_JDBC_URL -Dsonar.jdbc.username=$SONAR_JDBC_USERNAME -Dsonar.jdbc.password=$SONAR_JDBC_PASSWORD -Dsonar.log.level=DEBUG
MAVEN_OPTS: -XX:MaxPermSize=512m -Xmx8192m`
When I perform an analysis with Jenkins I get the following error:
<code>[INFO] [08:57:22.023] Store results in database
[DEBUG] [08:57:22.029] Execute org.******.batch.phases.GraphPersister
[DEBUG] [08:57:22.126] Execute org.******.batch.index.SourcePersister
[DEBUG] [08:57:28.263] Updating semaphore batch-com.mycompany.myapp:myapp
[DEBUG] [08:57:38.265] Updating semaphore batch-com.mycompany.myapp:myapp
[DEBUG] [08:57:48.267] Updating semaphore batch-com.mycompany.myapp:myapp
[DEBUG] [08:57:51.788] Execute org.******.batch.index.ResourcePersister
[DEBUG] [08:57:51.788] Execute org.******.batch.index.MeasurePersister
[DEBUG] [08:57:58.269] Updating semaphore batch-com.mycompany.myapp:myapp
[DEBUG] [08:58:08.271] Updating semaphore batch-com.mycompany.myapp:myapp
ERROR: Maven JVM terminated unexpectedly with exit code 137</code>
Can you help?
I had same issue before, but for me the lack of virtual memory of my linux instance was the issue.
Increasing the memory Maven options didn't help much
OpsCenter version: 5.1.0 and
DSE Version: 4.6.0
Creating a brand new cluster by using OpsCenter directly, gives us the following error. It randomly works with the same settings but 95% of the times it fails with the same error. Opscenter is running on its own box but sharing the same Security groups as the cluster instances. For good measure, I have opened up all TCP ports to all IPs. The following is the stack trace of the error from the opscenterd.log:
*2015-03-19 10:06:12+0000 [] INFO: Starting provisioning process
2015-03-19 10:06:12+0000 [] INFO: Starting installation phase of cluster provisioning
2015-03-19 10:06:13+0000 [] WARN: HTTP request http://10.x.x.x:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2015-03-19 10:06:13+0000 [] INFO: Beginning install of OpsCenter agent to 54.x.x.x
2015-03-19 10:06:26+0000 [] WARN: HTTP request http://10.x.x.x:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2015-03-19 10:06:31+0000 [] INFO: Agent for ip 10.x.x.x is version None
2015-03-19 10:06:31+0000 [] INFO: Agent for ip 10.x.x.x is version u'5.1.0'
2015-03-19 10:07:23+0000 [] INFO: Successfully installed agent and dse on node 10.x.x.x
2015-03-19 10:07:23+0000 [] INFO: Beginning "stop" phase of cluster provisioning
2015-03-19 10:07:25+0000 [] WARN: Marking request '10.x.x.x: /ops/stop' (f6708fa2-b45f-42b4-b992-90a82b460ac7) as failed: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] ERROR: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] WARN: Marking request 'stop stage' (0b6fcb6b-96ba-404e-a484-b4b6b167b309) as failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] ERROR: Stop stage failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] WARN: Marking request 'provision' (daf1c15d-92e3-40b0-83ca-34d548ea835b) as failed: Stop stage failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] ERROR:
2015-03-19 10:07:25+0000 [] ERROR: Cluster provisioning failed: Exception: Stop stage failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] ERROR: Failed to provision cluster: Cluster provisioning failed: Exception: Stop stage failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:25+0000 [] WARN: Marking request 28c021fd-d21a-4fed-bb5c-a4fe17d362e0 as failed: Cluster provisioning failed: Exception: Stop stage failed: Failed to stop node 10.x.x.x: /usr/sbin/service dse stop failed
exit status: 1
stdout:
log_daemon_msg is a shell function
Cassandra 2.0 and later require Java 7 or later.
2015-03-19 10:07:41+0000 [] WARN: Unable to find a matching cluster for node with IP [u'fe80:0:0:0:2000:aff:feeb:31c7%2', u'10.x.x.x', u'0:0:0:0:0:0:0:1%1', u'127.0.0.1']; the message was [u'5.1.0', u'/1947480708/conf']. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
Appreciate any help!
Thanks in advance
Harsha
OpCenter developer here. I make the OpsCenter provisioning features go zoom (or splat occasionally as you've seen). It is with sadness and shame that I must tell you that you're hitting a bug.
The Datastax AMI version 2.4 used by OpsCenter provisioning (https://github.com/riptano/ComboAMI/tree/2.4) does quite a bit of work at boot time via startup scripts. One of those tasks is to set up some gpg repository keys used to validate packages. Intermittently that process can fail, breaking package installs and leading to the series of errors that you're seeing. This failure is intermittent and has greatly increased in frequency recently. If you check /home/ubuntu/datastax-ami/ami.log you should see the gpg key failures that begin the rest of the failure chain.
Unfortunately, this error is pretty far down the technology stack and is difficult to manually work around. If you just need to provision a single cluster you can retry until you get a good run. Otherwise your best best is to manually launch the instances and use local provisioning to deploy dse/dsc to their private ip addresses:
Launch instances using ami-ada2b6c4 (assuming you're in us-east-1)
Make sure to add the instances to the OpsCenterSecurity group.
Make sure you have the private half of the keypair you use (you'll need it during local provisioning)
On the instance data page, hit the advanced pulldown and add the following userdata as text "--raidonly --java7"
Do a local-provisioning run against the private-ip's
Not a super-simple workaround. I wish your experience with OpsCenter this time around was more awesome. The good news is I'm on this bug and it will be fixed in an upcoming point release.
Edit: No longer necessary to manually remove /etc/security/limits.d/cassandra.conf
if its just complaining about java then install the java 7 preferably datastax wants oracle jdk and jre. you might already have java 7 and another version on your nodes but java 7 is not the default version. to change this do:
sudo update-java-alternatives -s java-7-oracle
which is a command you can script to run with ssh so you dont have to log in to each node
I've been trying to automate everything about our deployment and have had some reasonable incremental success until today, when it all went backwards.
.ebextensions/01settings.config
container_commands:
01seed:
command: rake db:seed
leader_only: true
This has occasionally run, but mostly does not. When it does on a deploy, I'll see something like:
2013-08-13 21:46:55,606 [INFO] Running configSets: Infra-EmbeddedPostBuild
2013-08-13 21:46:55,607 [INFO] Running configSet Infra-EmbeddedPostBuild
2013-08-13 21:46:55,609 [INFO] Running config postbuild_0_acme_development_02e6a3b5e2d179b50d766a8dc16f8243__ebextensions_01settings_config
2013-08-13 21:46:55,609 [DEBUG] No packages specified
2013-08-13 21:46:55,609 [DEBUG] No groups specified
2013-08-13 21:46:55,609 [DEBUG] No users specified
2013-08-13 21:46:55,609 [DEBUG] No sources specified
2013-08-13 21:46:55,609 [DEBUG] No files specified
2013-08-13 21:46:55,610 [DEBUG] Running command 01seed
2013-08-13 21:46:55,610 [DEBUG] Generating defaults for command 01seed
2013-08-13 21:46:55,810 [DEBUG] Running test for command 01seed
2013-08-13 21:46:55,839 [DEBUG] Test command output:
2013-08-13 21:46:55,840 [DEBUG] Test for command 01seed passed
2013-08-13 21:47:08,019 [INFO] Command 01seed succeeded
2013-08-13 21:47:08,020 [DEBUG] No services specified
2013-08-13 21:47:08,034 [INFO] ConfigSets completed
Other times, I'll deploy the same thing to the same instance, and that same section will read:
2013-08-13 22:15:15,360 [INFO] Running configSets: Infra-EmbeddedPostBuild
2013-08-13 22:15:15,362 [INFO] Running configSet Infra-EmbeddedPostBuild
2013-08-13 22:15:15,362 [INFO] ConfigSets completed
I can't tell you how frustrating this experience has been. I plan to share this automation for the rails deployment via rake as a gem very soon, but I'm hesitant because this won't work the same every time.
What are the reasons an .ebextensions/*.config wouldn't even be attempted?
EDIT:
It appears that my deep_symbolized_hash.to_yaml puts out an .ebextensions/01settings.config that looks like:
:container_commands:
:01seed:
:command: rake db:seed
:leader_only: true
So, while this is valid yaml, the elastic beanstalk configset running process of Infra-EmbeddedPostBuild just walks right by the my config without complaint.
Does that sound right? Are we really to hope that eb just swallows this?
This is confirmed. If using a ruby hash with keys that have been symbolized, you need to unsymbolize the keys before to_yaml or else post process and remove the leading ':' semi-colon.
Also confirmed, if eb doesn't like your .config file on deploy, it just swallows it and doesn't even log anything!