When I run my Dataflow pipeline, I get the exception below complaining that my DoFn can't be serialized. How do I fix this?
Here's the stack trace:
Caused by: java.lang.IllegalArgumentException: unable to serialize contrail.dataflow.AvroMRTransforms$AvroReducerDoFn#bba0fc2
at com.google.cloud.dataflow.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:51)
at com.google.cloud.dataflow.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:81)
at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.ensureSerializable(DirectPipelineRunner.java:784)
at com.google.cloud.dataflow.sdk.transforms.ParDo.evaluateHelper(ParDo.java:1025)
at com.google.cloud.dataflow.sdk.transforms.ParDo.evaluateSingleHelper(ParDo.java:963)
at com.google.cloud.dataflow.sdk.transforms.ParDo.access$000(ParDo.java:441)
at com.google.cloud.dataflow.sdk.transforms.ParDo$1.evaluate(ParDo.java:951)
at com.google.cloud.dataflow.sdk.transforms.ParDo$1.evaluate(ParDo.java:946)
at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:611)
at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:200)
at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:196)
at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:109)
at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:204)
at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:584)
at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:328)
at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:70)
at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
at contrail.stages.DataflowStage.stageMain(DataflowStage.java:51)
at contrail.stages.NonMRStage.execute(NonMRStage.java:130)
at contrail.stages.NonMRStage.run(NonMRStage.java:157)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at contrail.stages.ValidateGraphDataflow.main(ValidateGraphDataflow.java:139)
... 6 more
Caused by: java.io.NotSerializableException: org.apache.hadoop.mapred.JobConf
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at com.google.cloud.dataflow.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:47)
... 27 more
To add to what Jeremy says...
Another common cause of Serializable issues is when you use an anonymous DoFn within a non-static context. Anonymous inner classes have an implicit pointer to the enclosing class, which will cause it to get serialized as well.
If you scroll through the stack trace, one of the causes clearly identifies the data that isn't serializable.
Caused by: java.io.NotSerializableException: org.apache.hadoop.mapred.JobConf
The problem was my DoFn was taking a JobConf instance in the constructor and storing it in an instance variable. I was assuming JobConf was serializable but it turns out it isn't.
To solve this I did the following
I marked the JobConf member variable as transient so that it wouldn't be serialized.
I created a separate variable of type byte[] to store a serialized version of JobConf
In my constructor I serialized JobConf to a byte[] and stored it in an instance variable.
I overrode startBundle and deserialized the JobConf from the byte[]
Here's a gist with my DoFn.
Related
I am implementing a Pub/Sub to BigQuery pipeline. It looks similar to How to create read transform using ParDo and DoFn in Apache Beam, but here, I have already a PCollection created.
I am following what is described in the Apache Beam documentation to implement a ParDo operation to prepare a table row using the following pipeline:
static class convertToTableRowFn extends DoFn<PubsubMessage, TableRow> {
#ProcessElement
public void processElement(ProcessContext c) {
PubsubMessage message = c.element();
// Retrieve data from message
String rawData = message.getData();
Instant timestamp = new Instant(new Date());
// Prepare TableRow
TableRow row = new TableRow().set("message", rawData).set("ts_reception", timestamp);
c.output(row);
}
}
// Read input from Pub/Sub
pipeline.apply("Read from Pub/Sub",PubsubIO.readMessagesWithAttributes().fromTopic(topicPath))
.apply("Prepare raw data for insertion", ParDo.of(new convertToTableRowFn()))
.apply("Insert in Big Query", BigQueryIO.writeTableRows().to(BQTable));
I found the DoFn function in a gist.
I keep getting the following error:
The method apply(String, PTransform<? super PCollection<PubsubMessage>,OutputT>) in the type PCollection<PubsubMessage> is not applicable for the arguments (String, ParDo.SingleOutput<PubsubMessage,TableRow>)
I always understood that a ParDo/DoFn operations is a element-wise PTransform operation, am I wrong ? I never got this type of error in Python, so I'm a bit confused about why this is happening.
You're right, ParDos are element-wise transforms and your approach looks correct.
What you're seeing is the compilation error. Something like this happens when the argument type of the apply() method that was inferred by java compiler doesn't match the type of the actual input, e.g. convertToTableRowFn.
From the error you're seeing it looks like java infers that the second parameter for apply() is of type PTransform<? super PCollection<PubsubMessage>,OutputT>, while you're passing the subclass of ParDo.SingleOutput<PubsubMessage,TableRow> instead (your convertToTableRowFn). Looking at the definition of SingleOutput your convertToTableRowFn is basically a PTransform<PCollection<? extends PubsubMessage>, PCollection<TableRow>>. And java fails to use it in apply where it expects PTransform<? super PCollection<PubsubMessage>,OutputT>.
What looks suspicious is that java didn't infer the OutputT to PCollection<TableRow>. One reason it would fail to do so if you have other errors. Are you sure you don't have other errors as well?
For example, looking at convertToTableRowFn you're calling message.getData() which doesn't exist when I'm trying to do it and it fails compilation there. In my case I need to do something like this instead: rawData = new String(message.getPayload(), Charset.defaultCharset()). Also .to(BQTable)) expects a string (e.g. a string representing the BQ table name) as an argument, and you're passing some unknown symbol BQTable (maybe it exists in your program somewhere though and this is not a problem in your case).
After I fix these two errors your code compiles for me, apply() is fully inferred and the types are compatible.
I have a legacy system that I am migrating to Grails 3 but I'm running into a naming convention issue. One of our tables is called Application but that is also the name of the Groovy class that runs the application. There seems to be an issue where GORM is confusing the 2 classes and giving type-casting errors. I have tried explicitly casting my Application class in my ApplicationDegree class to try and force GORM to recognize which class I am trying to use but no success.
I know an option is that I can rename my domain Application class to something else and manually map it to the appropriate database table but I want to avoid that as I foresee issues down the road.
So, other than renaming my Domain classes, how can I get Grails/GORM to make the correct mapping.
Grails 3.1.4
Code:
ApplicationDegree ad = ApplicationDegree.findById(10);
Class:
class ApplicationDegree {
Boolean isPrimary
domain.package.Application application
Degree degree
static hasMany = [applicationDegreeMajors: ApplicationDegreeMajor,
applicationDegreeMinors: ApplicationDegreeMinor]
static belongsTo = [domain.package.Application, Degree]
static mapping = {
version false
degree column: 'degree_code'
isPrimary column: 'isPrimary'
}
}
StackTrace:
ERROR org.grails.spring.beans.factory.OptimizedAutowireCapableBeanFactory - Bean couldn't be autowired using grails optimization: Property 'application' threw exception; nested exception is java.lang.IllegalArgumentException: argument type mismatch
ERROR org.grails.spring.beans.factory.OptimizedAutowireCapableBeanFactory - Retrying using spring autowire
ERROR org.grails.web.errors.GrailsExceptionResolver - IllegalStateException occurred when processing request: [GET] /app4grad_V2/college/applications/
Cannot convert value of type [app4grad.Application] to required type [domain.package.Application] for property 'application': no matching editors or conversion strategy found. Stacktrace follows:
java.lang.reflect.InvocationTargetException: null
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.hibernate.InstantiationException: Could not instantiate entity: : domain.package.ApplicationDegree
at app4grad.college.ApplicationsController$$EPuFyMDe.index(ApplicationsController.groovy:14)
... 3 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
... 4 common frames omitted
Caused by: org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type [app4grad.Application] to required type [domain.package.Application] for property 'application'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [app4grad.Application] to required type [domain.package.Application] for property 'application': no matching editors or conversion strategy found
... 4 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot convert value of type [app4grad.Application] to required type [domain.package.Application] for property 'application': no matching editors or conversion strategy found
... 4 common frames omitted
I had the same problem, the problem is that you're using a reserved name : https://grails.org/wiki/Reserved%20Names
Using Intelij Idea you can easily rename you're domain class and the IDE will do the rest for you.
Using JSF 2 on JBoss AS 7
Getting the following error:
07:36:39,579 SEVERE [javax.enterprise.resource.webcontainer.jsf.application] (http-/172.20.91.126:12580-16)
Error Rendering View[/views/afgarendesok.xhtml]:
com.sun.faces.mgbean.ManagedBeanCreationException:
Unable to set property searchManager for managed bean afgArendeBacking
at com.sun.faces.mgbean.ManagedBeanBuilder$BakedBeanProperty.set(ManagedBeanBuilder.java:615)
The searchManager property is defined in the AfgArendeBacking class as:
#ManagedProperty(value="#{afgArendeSokManager}")
private AfgArendeSokManager searchManager;
#Override
public AfgArendeSokManager getSearchManager() {
return searchManager;
}
public void setSearchManager(AfgArendeSokManager searchManager) {
this.searchManager = searchManager;
}
The AfgArendeSokManager is a #ManagedBean that is #SessionScoped.
Two things I don't get. One is why the error shuts down all usage of JSF not just for the session producing the error. The error seems to appear after non-usage both below the default session timeout and beyond. The other odd this is that a null pointer exception at line 606 in the BakedBeanProperty has to be the one the writeMethod variable. That variable is created via the PropertyDescriptor.getWriteMethod() call. This should have bombed earlier when creating baked bean (i.e. bakeBeanProperty method).
Any ideas how to debug? The property "searchManager" is resolved correctly since we can use the JSF views normally (both the getter/setter exist).
The search manager is our session scratch pad for propagating stuff between view and request scoped backing beans.
The article explains the issue of using reflection to access methods with covariant return types (see here: https://dzone.com/articles/covariant-return-type-abyssal). The article relates to Java 6 but the background information is very useful.
The issue you're facing and that hit us just this week (using Java 1.7.0_40) is not one of EL but of the java.beans.Introspector.
After upgrading to JSF2 (probably) one special accessor in an .xhtml file generates IllegalArgumentExceptions but I can't really find out why. Running the app on my local JBoss (4.2.2) does not generate this exception but this can be related to a difference between debug and live data.
Following stack trace is generated that I could extract from the production server log:
Caused by: javax.el.ELException: /xy/xy-subtemplate1.xhtml #131,45 value="#{someClass.someProperty}": java.lang.IllegalArgumentException
at com.sun.facelets.el.TagValueExpression.setValue(TagValueExpression.java:101)
at javax.faces.component.UIInput.updateModel(UIInput.java:818)
... 36 more
Caused by: java.lang.IllegalArgumentException
at sun.reflect.GeneratedMethodAccessor1134.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.el.BeanELResolver.setValue(BeanELResolver.java:108)
at com.sun.faces.el.DemuxCompositeELResolver._setValue(DemuxCompositeELResolver.java:255)
at com.sun.faces.el.DemuxCompositeELResolver.setValue(DemuxCompositeELResolver.java:281)
at org.apache.el.parser.AstValue.setValue(AstValue.java:114)
at org.apache.el.ValueExpressionImpl.setValue(ValueExpressionImpl.java:249)
at com.sun.facelets.el.TagValueExpression.setValue(TagValueExpression.java:93)
... 37 more
someClass is being iterated over in a list and has the following methods to access someProperty:
public int getSomeProperty() {
return this.getSomeRelatedEnum().ordinal();
}
public void setSomeProperty( final int index) {
this.setSomeRelatedEnum( SomeRelatedEnum.fromOrdinal( index) );
}
How can this lead to the IllegalArgumentException mentioned above?
That can happen when the setter method after all expects a different argument type than the one which is provided from EL on.
I guess that it's related to the fact that hardcoded/unconverted numbers in EL are by default treated as long and not as int. Try either changing the int to be long, or providing an explicit integer converter on the input component like so <h:someInput converter="javax.faces.Integer">.
By the way, why don't you just get/set the enum itself directly? Getting/setting the enum by its ordinal is whacky.
I'm trying to convert a domain class into JSON.
def converter = null
try{
converter = events as JSON
} catch(e) {
log.error "error during conversion to JSON"
log.error e
}
return converter.toString()
I always get this error:
org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Executing action [findEvents] of controller
[com.geoadapta.geodata.DataAccessJsonController] caused exception:
java.lang.StackOverflowError
at java.lang.Thread.run(Thread.java:619)
Caused by: org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.StackOverflowError
... 1 more
Caused by: java.lang.StackOverflowError
at org.codehaus.groovy.util.AbstractConcurrentMap.getOrPut(AbstractConcurrentMap.java:20)
at grails.converters.JSON.value(JSON.java:188)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
// hundreds of times
at grails.converters.JSON.value(JSON.java:192)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
at grails.converters.JSON.convertAnother(JSON.java:160)
at grails.converters.JSON.value(JSON.java:192)
2010-06-23 16:21:28,390 [http-8080-1] DEBUG [/GeoAdaptaApp].[grails] - Disabling the response for futher output
This is the domain class: http://www.copypastecode.com/31527
The only problem I can see is the 'context' reference. But even if I set to 'null', I still get the stack overflow.
My converter is not propagating circular references.
I'd like to debug it and find out which property is causing the problem, but no luck so far.
Any hints?
Cheers
Sorry, would prefer to add this as a comment, but do not have the rep to add comments yet.
In your domain class it sounds like there are several classes that extend off this one, how are you populating the 'events' object? Since you say the context is set to null and that is the only relationship that's defined directly, a list of unintended subclasses in your results is one way that springs to mind for how you could get into such a deep recursion scenario.
To work around the immediate problem so you can do some debugging, have you tried calling:
converter.setRenderDomainClassRelations(false)