After upgrading jenkins from 2.263.4 to 2.346.3 I am facing issue that all job configs are not getting loaded / visible to Jenkins UI.
In Jenkins logs there are errors like below for them:
Failed Loading item deploy_system_spec_SingleServicecom.thoughtworks.xstream.converters.ConversionException: Validation failed
---- Debugging information ----
message : Validation failed
scriptId : The script is not allowed to be executed in a build, check its configuration!
at org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder$ConverterImpl.callback(ScriptlerBuilder.java:318)
Caused: jenkins.util.xstream.CriticalXStreamException:
---- Debugging information ----
cause-exception : com.thoughtworks.xstream.converters.ConversionException
cause-message : Validation failed
class : org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder
required-type : org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder
converter-type : hudson.util.XStream2$AssociatedConverterImpl
path : /project/properties/hudson.model.ParametersDefinitionProperty/parameterDefinitions/org.biouno.unochoice.ChoiceParameter/script/scriptlerBuilder
line number : 46
class[1] : org.biouno.unochoice.model.ScriptlerScript
required-type[1] : org.biouno.unochoice.model.ScriptlerScript
converter-type[1] : hudson.util.RobustReflectionConverter
class[2] : org.biouno.unochoice.ChoiceParameter
required-type[2] : org.biouno.unochoice.ChoiceParameter
class[3] : java.util.ArrayList
required-type[3] : java.util.ArrayList
converter-type[2] : hudson.util.RobustCollectionConverter
class[4] : hudson.model.ParametersDefinitionProperty
required-type[4] : hudson.model.ParametersDefinitionProperty
class[5] : hudson.util.CopyOnWriteList
required-type[5] : hudson.util.CopyOnWriteList
class[6] : hudson.model.FreeStyleProject
required-type[6] : hudson.model.FreeStyleProject
version : not available
-------------------------------
at org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder$ConverterImpl.callback(ScriptlerBuilder.java:323)
at org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder$ConverterImpl.callback(ScriptlerBuilder.java:307)
at hudson.util.XStream2$PassthruConverter.unmarshal(XStream2.java:535)
at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:501)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:117)
at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:202)
at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:178)
at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:501)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1421)
at hudson.util.XStream2.unmarshal(XStream2.java:189)
at hudson.util.XStream2.unmarshal(XStream2.java:160)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1399)
at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1293)
at hudson.XmlFile.read(XmlFile.java:165)
Caused: java.io.IOException: Unable to read /opt/jenkins/home/jobs/deploy_system_spec_SingleService/config.xml
at hudson.XmlFile.read(XmlFile.java:167)
at hudson.model.Items.load(Items.java:375)
at jenkins.model.Jenkins$13.run(Jenkins.java:3420)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1158)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
But in Manage Jenkins - Script Approval I am not seeing anything pending approval.
It is hard to say what exactly went wrong since you've made a bug jump between LTS versions. I would recommend to read upgrade guide at first and make backups at second.
Check the owner of jobs folder in jenkins home folder on the server which is running jenkins.
Good luck with docs)
I did found out the problem, basically all missing jobs were failing like this:
cause-message : Validation failed
class : org.jenkinsci.plugins.scriptler.builder.ScriptlerBuilder
i.e. related to scripler plugin. And with new Jenkins / scriptler plugin version they seems to be by default disabled:
so needed to go for each of the scriptler scripts and select like this there:
Related
I've messed up my Gerrit installation. So I want to re-init Gerrit.
I tried to do:
java -jar gerrit-2.15.1.war init -d /my_gerrit_destination
and Gerrit asked if I wanted to use my previous settings. Perfect!
But.. I've messed up some file in the /index-folder so my initialization failed.
Is it possible to clean all or some of the directories/files and Gerrit will still ask me if I wanted to use same configurations as before?
I ask because the person that holds some of the used passwords in the config, is on vacation.
EDIT:
1.I removed the index directory.
2.I ran the above init command again.
3.Gerrit FAILED to start
4.Checked in the error.log and followed instructions about re-index some directories but got an error posted in error.log:
[2018-07-04 14:39:39,989] [main] WARN com.google.gerrit.sshd.SshDaemon : Cannot format SSHD host key [EdDSA]: invalid key type
[2018-07-04 14:39:40,006] [main] WARN com.google.gerrit.server.config.GitwebCgiConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)
[2018-07-04 14:39:41,069] [main] INFO org.eclipse.jetty.util.log : Logging initialized #11300ms
[2018-07-04 14:39:41,161] [main] INFO com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 1339m
[2018-07-04 14:39:41,635] [main] INFO com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /opt/gerrit/plugins
[2018-07-04 14:39:41,745] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) No index versions for index 'groups' ready; run java -jar /opt/gerrit/bin/gerrit.war reindex --index groups
1 error
at com.google.gerrit.server.index.VersionManager.initIndex(VersionManager.java:173)
at com.google.gerrit.server.index.VersionManager.start(VersionManager.java:94)
at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:92)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:349)
at com.google.gerrit.pgm.Daemon.run(Daemon.java:256)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:223)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:119)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:63)
at Main.main(Main.java:24)
5.I tried to run java -jar /opt/gerrit/bin/gerrit.war reindex --index groups but I get
`fatal: not a Gerrit site: '.'
fatal: Perhaps you need to run init first?`
6.New init fails to start Gerrit.
1) Remove the index directory
rm -rf GERRIT-SITE/index
2) Run the reindex command
java -jar gerrit-VERSION.war reindex -d GERRIT-SITE
When schemaExport is executed using Gradle for a Grails 3.3 application, below warning is in the log though ddl.sql is created. According to Grails 3 documentation sitemesh.xml is removed so naturally the file wouldn't be available. Am I missing something?
Tools and version: Grails 3.3.0, Gradle 3.3, Hibnerate 5, Sybase ASE 15.7
2017-09-13 12:50:45.264 WARN --- [ main] o.s.mock.web.MockServletContext : Couldn't determine real path of resource class path resource [src/main/webapp/WEB-INF/sitemesh.xml]
java.io.FileNotFoundException: class path resource [src/main/webapp/WEB-INF/sitemesh.xml] cannot be resolved to URL because it does not exist
at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:187)
at org.springframework.core.io.AbstractFileResolvingResource.getFile(AbstractFileResolvingResource.java:49)
at org.springframework.mock.web.MockServletContext.getRealPath(MockServletContext.java:458)
at org.grails.web.sitemesh.Grails5535Factory.<init>(Grails5535Factory.java:78)
at org.grails.web.servlet.view.SitemeshLayoutViewResolver.loadSitemeshConfig(SitemeshLayoutViewResolver.java:105)
at org.grails.web.servlet.view.SitemeshLayoutViewResolver.init(SitemeshLayoutViewResolver.java:67)
at org.grails.web.servlet.view.SitemeshLayoutViewResolver.onApplicationEvent(SitemeshLayoutViewResolver.java:146)
at org.grails.web.servlet.view.SitemeshLayoutViewResolver.onApplicationEvent(SitemeshLayoutViewResolver.java:42)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:883)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at grails.boot.GrailsApp.run(GrailsApp.groovy:83)
at grails.ui.command.GrailsApplicationContextCommandRunner.run(GrailsApplicationContextCommandRunner.groovy:55)
at grails.ui.command.GrailsApplicationContextCommandRunner.main(GrailsApplicationContextCommandRunner.groovy:102)
This is just a WARN message. The underlying spring class is looking for it and just saying it's not there. The default the log level does not show this output.
You can safely ignore the message or adjust your logger to not show it.
https://github.com/spring-projects/spring-framework/blob/master/spring-test/src/main/java/org/springframework/mock/web/MockServletContext.java#L460
I have Groovy and Grails project, and I have used the asset-pipeline.
Grails: 3.0.17 # Gradle: 2.12
In my build.gradle file I have applied asset-pipeline as below :
apply plugin: "asset-pipeline"
While building the project i am getting following error :
org.gradle.api.GradleScriptException: A problem occurred evaluating project ':ciab-base'.
at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:154)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:59)
at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:492)
at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:86)
at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:47)
at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:35)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:129)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:80)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:36)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:51)
at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:169)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
at org.gradle.launcher.Main.doAction(Main.java:33)
at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:127)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:56)
Caused by: org.gradle.api.internal.plugins.PluginApplicationException: Failed to apply plugin [id 'asset-pipeline']
at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:147)
at org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:102)
at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyType(DefaultObjectConfigurationAction.java:113)
at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$200(DefaultObjectConfigurationAction.java:36)
at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$3.run(DefaultObjectConfigurationAction.java:80)
at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:136)
at org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:46)
at org.gradle.api.plugins.PluginAware$apply.call(Unknown Source)
at org.gradle.api.internal.project.ProjectScript.apply(ProjectScript.groovy:34)
at org.gradle.api.Script$apply$0.callCurrent(Unknown Source)
at build_eutmpfkdca3gfie59pyomub8f.run(/home/urvin/Urvin/Repo/ciab-plugin/ciab-base/build.gradle:32)
at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:52)
... 35 more
Caused by: groovy.lang.MissingPropertyException: No such property: COMPILE_ONLY_CONFIGURATION_NAME for class: org.gradle.api.plugins.JavaPlugin
at asset.pipeline.gradle.AssetPipelinePlugin$_createGradleConfiguration_closure2.doCall(AssetPipelinePlugin.groovy:191)
at org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:67)
at org.gradle.internal.Actions$FilteredAction.execute(Actions.java:201)
at org.gradle.api.internal.DefaultDomainObjectCollection.all(DefaultDomainObjectCollection.java:110)
at org.gradle.api.internal.DefaultDomainObjectCollection.all(DefaultDomainObjectCollection.java:115)
at org.gradle.api.internal.DefaultDomainObjectCollection.withType(DefaultDomainObjectCollection.java:126)
at org.gradle.api.DomainObjectCollection$withType$0.call(Unknown Source)
at asset.pipeline.gradle.AssetPipelinePlugin.createGradleConfiguration(AssetPipelinePlugin.groovy:189)
at asset.pipeline.gradle.AssetPipelinePlugin.apply(AssetPipelinePlugin.groovy:52)
at asset.pipeline.gradle.AssetPipelinePlugin.apply(AssetPipelinePlugin.groovy)
at org.gradle.api.internal.plugins.ImperativeOnlyPluginApplicator.applyImperative(ImperativeOnlyPluginApplicator.java:35)
at org.gradle.api.internal.plugins.RulesCapablePluginApplicator.applyImperative(RulesCapablePluginApplicator.java:42)
at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:133)
... 46 more
I had the same error.
This is due to a mismatch between the version of grails you're using, expecting gradle 2.11, and the asset pipeline plugin, which needs 2.12.
To solve it you can start grails with (replace by your path to gradle):
$ GRAILS_GRADLE_HOME=~/.sdkman/candidates/gradle/2.12/ grails
Some of our Dataflow jobs randomly crash while reading source data files.
The following error is written in the job logs (there is nothing in the workers logs) :
11 févr. 2016 à 08:30:54
(33b59f945cff28ab): Workflow failed.
Causes: (fecf7537c059fece): S02:read-edn-file2/TextIO.Read+read-edn-file2
/ParDo(ff19274a)+ParDo(ff19274a)5+ParDo(ff19274a)6+RemoveDuplicates
/CreateIndex+RemoveDuplicates/Combine.PerKey
/GroupByKey+RemoveDuplicates/Combine.PerKey/Combine.GroupedValues
/Partial+RemoveDuplicates/Combine.PerKey/GroupByKey/Reify+RemoveDuplicates
/Combine.PerKey/GroupByKey/Write faile
We also get that kind of error sometimes (logged in the workers logs) :
2016-02-15T10:27:41.024Z: Basic: S18: (43c8777b75bc373e): Executing operation group-by2/GroupByKey/Read+group-by2/GroupByKey/GroupByWindow+ParDo(ff19274a)19+ParDo(ff19274a)20+ParDo(ff19274a)21+write-edn-file3/ParDo(ff19274a)+write-bq-table-from-clj3/ParDo(ff19274a)+write-bq-table-from-clj3/BigQueryIO.Write+write-edn-file3/TextIO.Write
2016-02-15T10:28:03.994Z: Error: (af73c53187b7243a): java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 503,
"errors" : [ {
"domain" : "global",
"message" : "Backend Error",
"reason" : "backendError"
} ],
"message" : "Backend Error"
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:431)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:289)
at com.google.cloud.dataflow.sdk.runners.worker.TextSink$TextFileWriter.close(TextSink.java:243)
at com.google.cloud.dataflow.sdk.util.common.worker.WriteOperation.finish(WriteOperation.java:100)
at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:254)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:191)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:144)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:180)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:161)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:148)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The source data files are stored in google cloud storage.
Data paths are correct and the job generally works after relaunching it.
We didn't experience this issue until the end of january.
Jobs are launched with these parameters :
--tempLocation='gstoragelocation' --stagingLocation='another gstorage location' --runner=BlockingDataflowPipelineRunner --numWorkers='a few dozen' --zone=europe-west1-d
SDK version : 1.3.0
Thanks
As a clearly-marked "backend error", this should be reported on either the Cloud Dataflow Public Issue Tracker](google-cloud-dataflow) or the more general Cloud Platform Public Issue Tracker and there's relatively little anybody on Stack Overflow could do to help you debug this.
I have a Ruby Stack configured with AWS Elastic Beanstalk. Last week, all my deployments were ok, but since I tried to change my MySQL RDS DB to use UTF-8, I got this problem each time I run an eb deploy :
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
ERROR: UnicodeEncodeError :: 'ascii' codec can't encode character u'\u2018' in position 90: ordinal not in range(128)
When I check on the AWS EB interface, I got this error message :
[Instance: i-2a4575cc] Command failed on instance. Return code: 1 Output: cp: cannot stat ‘/var/tmp/backup/10_bundle_install.sh’: No such file or directory. Hook /opt/elasticbeanstalk/hooks/appdeploy/post/05_revert_bundle_install_local.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
Finally, I checked inside the /var/log/eb-activity.log as mentionned above, and the last lines are the following :
[2015-06-16T09:03:39.079Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2015-06-16T09:03:39.079Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2015-06-16T09:03:39.079Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1/AppDeployPostHook/05_revert_bundle_install_local.sh] : Starting activity...
[2015-06-16T09:03:39.088Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1/AppDeployPostHook/05_revert_bundle_install_local.sh] : Activity execution failed, because: cp: cannot stat ‘/var/tmp/backup/10_bundle_install.sh’: No such file or directory (ElasticBeanstalk::ExternalInvocationError)
caused by: cp: cannot stat ‘/var/tmp/backup/10_bundle_install.sh’: No such file or directory (Executor::NonZeroExitStatus)
[2015-06-16T09:03:39.088Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1/AppDeployPostHook/05_revert_bundle_install_local.sh] : Activity failed.
[2015-06-16T09:03:39.088Z] INFO [23525] - [CMD AppDeploy/AppDeployStage1/AppDeployPostHook] : Activity failed.
[2015-06-16T09:03:39.088Z] INFO [23525] - [CMD-AppDeploy/AppDeployStage1] : Activity failed.
[2015-06-16T09:03:39.088Z] INFO [23525] - [CMD-AppDeploy] : Completed activity. Result:
Command CMD-AppDeploy failed.
The problem seems to come from the cp command requested in the 05_revert_bundle_install_local.sh script :
cp -a /var/tmp/backup/10_bundle_install.sh /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh
The /var/tmp/backup/10_bundle_install.sh doesn't exist.
Do you know what could have happened ? And how can I solve it to avoid the error message and make clean deployments ?
Thank you !
Have a look here.
Apparently when installing the default sample app, beanstalk also installs this file and "forgets" to delete it. So the solution is simply to delete it manually.