kafka streams stateful mode is error (rocksdb don't initialize) - docker

error code :
Failed to close task manager due to the following error:
java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.DBOptions
at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:133)
at org.apache.kafka.streams.state.internals.TimestampedSegment.openDB(TimestampedSegment.java:49)
at org.apache.kafka.streams.state.internals.TimestampedSegments.getOrCreateSegment(TimestampedSegments.java:50)
at org.apache.kafka.streams.state.internals.TimestampedSegments.getOrCreateSegment(TimestampedSegments.java:25)
at org.apache.kafka.streams.state.internals.AbstractSegments.getOrCreateSegmentIfLive(AbstractSegments.java:84)
at org.apache.kafka.streams.state.internals.AbstractRocksDBSegmentedBytesStore.put(AbstractRocksDBSegmentedBytesStore.java:146)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:61)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:27)
at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:111)
at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:34)
at org.apache.kafka.streams.state.internals.CachingWindowStore.putAndMaybeForward(CachingWindowStore.java:106)
at org.apache.kafka.streams.state.internals.CachingWindowStore.lambda$initInternal$0(CachingWindowStore.java:86)
at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:151)
at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:109)
at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:124)
at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:291)
at org.apache.kafka.streams.state.internals.WrappedStateStore.flush(WrappedStateStore.java:84)
at org.apache.kafka.streams.state.internals.MeteredWindowStore.lambda$flush$4(MeteredWindowStore.java:200)
at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:801)
at org.apache.kafka.streams.state.internals.MeteredWindowStore.flush(MeteredWindowStore.java:200)
at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:282)
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(StreamTask.java:647)
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:745)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.closeTask(AssignedStreamsTasks.java:81)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.closeTask(AssignedStreamsTasks.java:37)
at org.apache.kafka.streams.processor.internals.AssignedTasks.shutdown(AssignedTasks.java:256)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.shutdown(AssignedStreamsTasks.java:535)
at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(TaskManager.java:292)
at org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(StreamThread.java:1133)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:682)
Local works fine.
In the environment where the error occurs, the error occurs when uploading it to docker.
kafka streams version : 2.5.0

Related

A command from the RDF4J framework is throwing an error

When running the following commands, an error is thrown by the add command. The rdf4j framework is used for communicating with a graphdb Knowledge Base:
import org.eclipse.rdf4j.model.Model;
Model model;
//statements have been added to the model
public RepositoryConnection connection;
...
connection.add(model); <-- error is thrown
An error is thrown in add command
2022-01-24 09:46:02 [http-nio-8080-exec-5] ERROR restapi.SubmitRMService - unable to rollback transaction. HTTP error code 404
org.eclipse.rdf4j.repository.RepositoryException: unable to rollback transaction. HTTP error code 404
at org.eclipse.rdf4j.http.client.RDF4JProtocolSession.rollbackTransaction(RDF4JProtocolSession.java:786)
at org.eclipse.rdf4j.repository.http.HTTPRepositoryConnection.rollback(HTTPRepositoryConnection.java:354)
at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.conditionalRollback(AbstractRepositoryConnection.java:335)
at org.eclipse.rdf4j.repository.base.AbstractRepositoryConnection.add(AbstractRepositoryConnection.java:379)
The code is running successfully in two other environments except one we are setting with kubernetes. We ensured that /opt/graphdb/home has write permissions.
Question: I just do not understand the 404 RepositoryException error. The code can successfully run queries before the .add(model), so the connection with the graphdb is ok.

Failed to start the VM error when starting a Dataflow SQL job

Getting the following error when I try to launch a Dataflow SQL job:
Failed to start the VM, launcher-____, used for launching because of status code: INVALID_ARGUMENT, reason: Error: Message: Invalid value for field 'resource.networkInterfaces[0].network': 'global/networks/default'. The referenced network resource cannot be found. HTTP Code: 400.
This issue just started today.
Adding the default network solved the issue.

Flink consumer job is unable to connect to Kafka from Docker

I have a simple streaming Flink Scala job which connects to a Kafka topic and maps its
org.apache.avro.generic.GenericRecord messages and map into Json string.
When it is running in IntelliJ it ingests the topic well and printing out the jsons.
When I run it in docker-compose I got the following exception:
com.esotericsoftware.kryo.KryoException: Error constructing instance of class: org.apache.avro.Schema$LockableArrayList
Serialization trace:
types (org.apache.avro.Schema$UnionSchema)
schema (org.apache.avro.Schema$Field)
fieldMap (org.apache.avro.Schema$RecordSchema)
schema (org.apache.avro.generic.GenericData$Record)
at com.twitter.chill.Instantiators$$anon$1.newInstance(KryoBase.scala:136)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1061)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.create(CollectionSerializer.java:89)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:93)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:657)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:262)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:111)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:37)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:635)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:612)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:592)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:727)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:705)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.emitRecord(KafkaFetcher.java:185)
at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:150)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:715)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:208)
Caused by: java.lang.IllegalAccessException: Class com.twitter.chill.Instantiators$ can not access a member of class org.apache.avro.Schema$LockableArrayList with modifiers "public"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
at java.lang.reflect.Constructor.newInstance(Constructor.java:413)
at com.twitter.chill.Instantiators$.$anonfun$normalJava$1(KryoBase.scala:170)
at com.twitter.chill.Instantiators$$anon$1.newInstance(KryoBase.scala:133)
... 37 more
I tried forcing Avro serialization with:
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
env.getConfig.disableForceKryo()
env.getConfig.enableForceAvro()
but got the same error.
Based on [this][1] I'm using all Flink related dependencies as "provided" with no good result.
What can be the difference between running the job in the IDE and in
Docker?
How can I fix the job to be able to read the Kafka topic from
Docker?
How shall I setup Docker for this?
What can I handle Kryo/Avro serialization issue?
SS
[1]: http://www.alternatestack.com/development/com-esotericsoftware-kryo-kryoexception-unusual-solution-upgrading-flink/

Error 401 on AdminApp deployment to Websphere 9 Docker

While running jython py script with adminApp.install to install on Docker WAS, I'm getting the Error:
WASX7017E: Exception received while running file
"deployWebSphere_with_install_standAlone.py"; exception information:
com.ibm.websphere.management.filetransfer.client.TransferFailedException:
401 Unauthorized (for:
C:\Users\devopsjava\AppData\Local\Temp\app5674632847255103189.ear).
I tried give full control on this path,
I tried set custom property:com.ibm.websphere.management.filetransfer.serverBasicAuth=true
(https://www-01.ibm.com/support/docview.wss?uid=swg1PK71800)
My code:
options = ['-node ',node,'-cell ',cell,"-server ",server]
print AdminApp.install(packageFile, options)
I always get :
WASX7017E: Exception received while running file
"deployWebSphere_with_install_standAlone.py"; exception information:
com.ibm.websphere.management.filetransfer.client.TransferFailedException:
401 Unauthorized (for:
C:\Users\devopsjava\AppData\Local\Temp\app5674632847255103189.ear).
Maybe, I need to set some more property, or there is problem with security between docker domain and the outside world ?

Error while starting grails

I am trying to work on grails through command prompt.
I am using the below versions:
| Grails Version: 3.2.3
| Groovy Version: 2.4.7
| JVM Version: 1.8.0_112
The following error is thrown:
| Error Error occurred running Grails CLI: This is usually a temporary
error dur ing hostname resolution and means that the local server did
not receive a respon se from an authoritative server (repo.grails.org)
(Use --stacktrace to see the f ull trace)
Upon using stacktrace the following is displayed:
| Error Error occurred running Grails CLI: This is usually a temporary
error dur ing hostname resolution and means that the local server did
not receive a respon se from an authoritative server (repo.grails.org)
(NOTE: Stack trace has been fi ltered. Use --verbose to see entire
trace.)
java.net.UnknownHostException: This is usually a temporary error
during hostname resolution and means that the local server did not
receive a response from an a uthoritative server (repo.grails.org)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefa
ultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHost
name(DefaultClientConnectionOperator.java:278)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnect
ion(DefaultClientConnectionOperator.java:162)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedCli
entConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(Default
RequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultReq
uestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:906)
at org.apache.http.impl.client.DecompressingHttpClient.execute(Decompres
singHttpClient.java:137)
at org.eclipse.aether.transport.http.HttpTransporter.execute(HttpTranspo
rter.java:287)
at org.eclipse.aether.transport.http.HttpTransporter.implGet(HttpTranspo
rter.java:243)
at org.eclipse.aether.spi.connector.transport.AbstractTransporter.get(Ab
stractTransporter.java:59)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRu
nner.runTask(BasicRepositoryConnector.java:447)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunne
r.run(BasicRepositoryConnector.java:350)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(Runn
ableErrorForwarder.java:67)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExe
cutor.execute(BasicRepositoryConnector.java:581)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(Basic
RepositoryConnector.java:249)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownl
oads(DefaultArtifactResolver.java:520)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(Defa
ultArtifactResolver.java:421)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtif
acts(DefaultArtifactResolver.java:246)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtif
act(DefaultArtifactResolver.java:223)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.
loadPom(DefaultArtifactDescriptorReader.java:320)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.
readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.resolveCa
chedArtifactDescriptor(DefaultDependencyCollector.java:535)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.getArtifa
ctDescriptorResult(DefaultDependencyCollector.java:519)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDe
pendency(DefaultDependencyCollector.java:409)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDe
pendency(DefaultDependencyCollector.java:363)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(D
efaultDependencyCollector.java:351)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDe
pendencies(DefaultDependencyCollector.java:254)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDepen
dencies(DefaultRepositorySystem.java:341)
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.resolve
(AetherGrapeEngine.java:302)
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.resolve
(AetherGrapeEngine.java:284)
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.resolve
(AetherGrapeEngine.java:276)
at org.grails.cli.boot.GrailsDependencyVersions.<init>(GrailsDependencyV
ersions.groovy:53)
at org.grails.cli.boot.GrailsDependencyVersions.<init>(GrailsDependencyV
ersions.groovy:49)
at org.grails.cli.profile.repository.MavenProfileRepository.<init>(Maven
ProfileRepository.groovy:53)
at org.grails.cli.GrailsCli.createMavenProfileRepository(GrailsCli.groov
y:333)
at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:234)
at org.grails.cli.GrailsCli.main(GrailsCli.groovy:159)
| Error Error occurred running Grails CLI: This is usually a temporary
error dur ing hostname resolution and means that the local server did
not receive a respon se from an authoritative server (repo.grails.org)
Kindly help me resolve the above issue.
Most likely reason is your internet is not working or your are behind a proxy. See http://docs.grails.org/latest/guide/conf.html#proxyConfig for how to configure your proxy settings

Resources