Akka 2.5 Distributed Data on Docker + Alpine Linux - docker

After upgrading a service that uses Akka + Akka cluster sharding to the newly released Akka (2.5.0), we started encountering issues starting the system in Docker + Alpine Linux. From what I can infer, Akka Cluster sharding is configured to used Akka Distributed Data (which is not experimental anymore as of 2.5.0) and it is using LMDB (which requires GCC + glibc and it is not available in Alpine Linux).
My questions are as follows:
1) Is there any standard alternative supported by Akka instead of LMDB?
2) Is there any way to get LMDB to work in Alpine Linux?
Stack Trace:
[ERROR] [04/20/2017 13:42:19.014] [lotus-akka.actor.default-dispatcher-5] [akka://lotus/system/sharding/replicator/durableStore] Error relocating /tmp/lmdbjava-native-library-5972006786989102785.so: __fprintf_chk: symbol not found
akka.actor.ActorInitializationException: akka://lotus/system/sharding/replicator/durableStore: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:191)
at akka.actor.ActorCell.create(ActorCell.scala:600)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:454)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:476)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
at akka.dispatch.Mailbox.run(Mailbox.scala:223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at akka.util.Reflect$.instantiate(Reflect.scala:65)
at akka.actor.ArgsReflectConstructor.produce(IndirectActorProducer.scala:96)
at akka.actor.Props.newActor(Props.scala:213)
at akka.actor.ActorCell.newActor(ActorCell.scala:555)
at akka.actor.ActorCell.create(ActorCell.scala:581)
... 7 more
Caused by: java.lang.UnsatisfiedLinkError: Error relocating /tmp/lmdbjava-native-library-5972006786989102785.so: __fprintf_chk: symbol not found
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:70)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:49)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:59)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:158)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:89)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:43)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:325)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:304)
at org.lmdbjava.Library.<clinit>(Library.java:95)
at org.lmdbjava.Env$Builder.open(Env.java:406)
at org.lmdbjava.Env$Builder.open(Env.java:430)
at akka.cluster.ddata.LmdbDurableStore.<init>(DurableStore.scala:131)
... 16 more

Finally managed to solve this problem. Cluster sharding attempts to use durable storage by default (default is LMDB). For cluster sharding without using remember-entities, durable storage is not required.
Hence, the solution to this was to disable durable storage for cluster sharding by adding the following configuration
akka.cluster.sharding.distributed-data.durable.keys = []

Related

ThingsBoard installation failed

Starting ThingsBoard Installation...
Installing DataBase schema for entities...
Installing SQL DataBase schema part: schema-entities.sql
Unexpected error during ThingsBoard installation!
org.postgresql.util.PSQLException: The authentication type 10 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it is using an authentication scheme supported by the driver.
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:634)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:217)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:52)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:216)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.thingsboard.server.service.install.SqlAbstractDatabaseSchemaService.createDatabaseSchema(SqlAbstractDatabaseSchemaService.java:66)
at org.thingsboard.server.service.install.SqlAbstractDatabaseSchemaService.createDatabaseSchema(SqlAbstractDatabaseSchemaService.java:57)
at org.thingsboard.server.install.ThingsboardInstallService.performInstall(ThingsboardInstallService.java:194)
at org.thingsboard.server.ThingsboardInstallApplication.main(ThingsboardInstallApplication.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:597)
Unexpected error during ThingsBoard installation!
ThingsBoard installation failed!
Please help someone! I'm trying to install thingsBoard but installation failed, I have pasted the print from cmd.
Thanks.
Refer to: https://github.com/thingsboard/thingsboard/issues/3667.
Download from https://jdbc.postgresql.org/download.html -> Other Versions the driver file:
postgresql-42.2.18.jar
Copy it to: c:/program files/AdoptOpenJDK/jdk-8.0.282.8-hotspot/jre/lib/ext/.
I had a similar problem and found that postgre version 13 didn't support thingsboard. switching to postgre version 11 fixed the issue.

Dataflow Batch Job fails with "Failed to close some writers"

I am running a batch pipeline with the Apache Beam 2.2 SDK via the Cloud Dataflow service. There are 751 text files that I parse using TextIO.readAll() transform, deserialize and write to a date partitioned table in BigQuery.
First thing I noticed is that autoscaling was not really kicking in and left the pipeline at 15 workers, even though I was able to push throughput a lot higher when for example manually setting the number of workers to 250.
My pipeline fails with the following stack trace:
(abed94a6f5139e21): java.io.IOException: Failed to close some writers
at org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles.finishBundle(WriteBundlesToFiles.java:248)
Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
Service Unavailable
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:431)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:289)
at org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter.close(TableRowWriter.java:81)
at org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles.finishBundle(WriteBundlesToFiles.java:242)
at org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles$DoFnInvoker.invokeFinishBundle(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:187)
at com.google.cloud.dataflow.worker.SimpleParDoFn.finishBundle(SimpleParDoFn.java:407)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:60)
at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
at com.google.cloud.dataflow.worker.DataflowWorker.executeWork(DataflowWorker.java:330)
at com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:302)
at com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:251)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
Service Unavailable
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:357)
... 4 more
Should I try with even more workers or split the work across several pipelines?
Thanks to the comment by jkff it worked flawlessly - after setting --maxNumWorkers=250 (15 seems to be the standard maximum).
The error was a transient error that Dataflow would retry several times and in the end, the pipeline ran successfully.

Cassandra Digest Mismatch Error

I see the following message in Cassandra's debug.log frequently and sometimes before losing nodes in the cluster. Any ideas on what the message means, and how to fix the underlying issue?
DEBUG [ReadRepairStage:9346] 2017-11-06 22:29:46,135 ReadCallback.java:242 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-8713145541289520569, 00114c65616465722f6d61737465722f352e3100000364633100) (408c7e13eea38efc9429366038cbe4a3 vs 8ce8acece0966903ac590d3229099398)
at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) ~[cassandra-all-3.11.0.1900.jar:3.11.0.1900]
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) ~[cassandra-all-3.11.0.1900.jar:3.11.0.1900]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [cassandra-all-3.11.0.1900.jar:3.11.0.1900]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
Here are the details of the Cassandra cluster:
4 node cluster
Each is an AWS instance of type m4.2xlarge
Each has an io1 volume with 20000 IOPS
All on same VPC, with 10.0.0.x private IP addresses
DataStax Enterprise Server 5.1.5
I think these are harmless messages from read repair noticing different data on different nodes, and probably not the cause of your node going down. See a more detailed answer this question last year: Datastax Mismatch for Key Issue

Apache Ignite cache operation failure

I've installed Ignite using a docker image from docker hub. Ignite server node starts correctly. But I get the following exception when trying to update cache:
[SEVERE][rest-#35%null%][GridCacheCommandHandler] Failed to execute cache command: GridRestCacheRequest [cacheName=null, cacheFlags=0, ttl=null, super=GridRestRequest [destId=null, clientId=466b7ff5-c303-452e-8f2d-97d59c753de5, addr=null, cmd=CACHE_PUT]]
class org.apache.ignite.IgniteCheckedException: Failed to find cache for given cache name (null for default cache): null
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.localCache(GridCacheCommandHandler.java:754)
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.executeCommand(GridCacheCommandHandler.java:677)
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.handleAsync(GridCacheCommandHandler.java:468)
at org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:264)
at org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:87)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:153)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[14:57:18,637][SEVERE][rest-#35%null%][GridRestProcessor] Failed to handle request: CACHE_PUT
class org.apache.ignite.IgniteCheckedException: Failed to find cache for given cache name (null for default cache): null
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.localCache(GridCacheCommandHandler.java:754)
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.executeCommand(GridCacheCommandHandler.java:677)
at org.apache.ignite.internal.processors.rest.handlers.cache.GridCacheCommandHandler.handleAsync(GridCacheCommandHandler.java:468)
at org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:264)
at org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:87)
at org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:153)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Any ideas what is wrong?
You should create cache before you will start using it. Use getOrCreateCache method.
You could read more information in doc and check this example which use cache api.
Also, there are a lot examples in apache ignite for various use cases.

Embedded neo4j with tinkerpop - Same Instance,Multiple access

I see that 'blueprints-neo4j-graph-2.5.0' is the latest released version to use tinkerpop blueprints api with neo4j graph which is version 1.9.6.
A few questions on this:
I understand that this neo4j instance will be an embedded database with community version. Is that right? If so, how do I switch to enterprise version?
I tried including this as gradle dependency: compile group: 'org.neo4j', name: 'neo4j-enterprise', version:'2.0.1'
But, I think 'blueprints-neo4j-graph-2.5.0' will get its own version(neo4j 1.9.6) as a dependency.
Is it possible to use neo4j webadmin console with the embedded version?
I tried changing the database location in neo4j.server.properties to the same location as given in the configuration for embedded instance, but I am not able to access them simultaneously.
I made this change in my build.gradle.
compile group: 'com.tinkerpop.blueprints', name: 'blueprints-neo4j2-graph', version:'2.5.0'
This picks up neo4j 2.0.1.
Also, I changed the tinkerpopConfig ( I am using GraphFactory to open and close the graph) as mentioned in tinkerpop sample to be:
"blueprints.neo4jha.directory" : "C:/data/my-test-model-graph",
"blueprints.graph" : "com.tinkerpop.blueprints.impls.neo4j2.Neo4j2HaGraph",
"blueprints.neo4jha.conf.server_id" : "1",
"blueprints.neo4jha.conf.server" : "localhost:6364",
"blueprints.neo4jha.conf.cluster_server" : "localhost:5002",
"blueprints.neo4jha.conf.initial_hosts" : "localhost:5001,localhost:5002"
Now when I try to deploy, I get a null graph.
When you said, HA implementation requires Enterprise edition:
I also tried doing it this way,
compile(group: 'com.tinkerpop.blueprints', name: 'blueprints-neo4j2-graph', version:'2.5.0'){
exclude(module: 'neo4j')
exclude(module: 'neo4j-ha')
exclude(module: 'neo4j-management')
}
compile group: 'org.neo4j', name: 'neo4j-enterprise', version:'2.0.1'
which will get neo4j-enterprise-2.0.1.
But this also doesn't seem to work and is not able to open a graph instance.
Do you see anything odd here?
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.tinkerpop.blueprints.impls.neo4j2.Neo4j2HaGraph].
at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50) ~[blueprints-core-2.5.0.jar:na]
at com.repo.GraphManager.getGraphDatabase(GraphManager.java:314) [com.repo~repo_module~1.0.0-SNAPSHOT/:na]
at com.repo.GraphManager.start(GraphManager.java:164) [com.repo~repo_module~1.0.0-SNAPSHOT/:na]
at org.vertx.java.platform.Verticle.start(Verticle.java:82) [vertx-platform-2.1M1.jar:na]
at com.englishtown.vertx.hk2.HK2VerticleLoader.start(HK2VerticleLoader.java:77) [com.englishtown~vertx-mod-hk2~1.6.0-SNAPSHOT/:na]
at org.vertx.java.platform.impl.DefaultPlatformManager$19.run(DefaultPlatformManager.java:1551) [vertx-platform-2.1M1.jar:na]
at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:176) [vertx-core-2.1M1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:43) ~[blueprints-core-2.5.0.jar:na]
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1124) ~[na:1.7.0_51]
at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1162) ~[na:1.7.0_51]
at org.neo4j.kernel.configuration.Config.applyChanges(Config.java:122) ~[neo4j-kernel-2.0.1.jar:2.0.1]
at org.neo4j.kernel.configuration.Config.<init>(Config.java:88) ~[neo4j-kernel-2.0.1.jar:2.0.1]
at org.neo4j.kernel.InternalAbstractGraphDatabase.<init>(InternalAbstractGraphDatabase.java:278) ~[neo4j-kernel-2.0.1.jar:2.0.1]
at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphDatabase.java:136) ~[neo4j-ha-2.0.1.jar:2.0.1]
at org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory$1.newDatabase(HighlyAvailableGraphDatabaseFactory.java:47) ~[neo4j-ha-2.0.1.jar:2.0.1]
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198) ~[neo4j-kernel-2.0.1.jar:2.0.1]
at com.tinkerpop.blueprints.impls.neo4j2.Neo4j2HaGraph.<init>(Neo4j2HaGraph.java:23) ~[blueprints-neo4j2-graph-2.5.0.jar:na]
at com.tinkerpop.blueprints.impls.neo4j2.Neo4j2HaGraph.<init>(Neo4j2HaGraph.java:31) ~[blueprints-neo4j2-graph-2.5.0.jar:na]
... 14 common frames omitted
If you want Neo4j 2.x then you need to use this dependency:
<dependency>
<groupId>com.tinkerpop.blueprints</groupId>
<artifactId>blueprints-neo4j2-graph</artifactId>
<version>2.5.0</version>
</dependency>
As this version runs in embedded mode only one JVM process at a time can access the database, so you can't access a database via TinkerPop and have Neo4j Server running at the same time to get the Neo4j WebAdmin Console. If you want to run in that mode then you should use Neo4jHaGraph:
https://github.com/tinkerpop/blueprints/wiki/Neo4jHa-Implementation
which requires Enterprise Edition:
http://docs.neo4j.org/chunked/stable/ha-setup-tutorial.html
Neo4jHaGraph should also exist in the blueprints-neo4j2-graph packaging.

Resources