Taking data from twitter and load it to hdfs using Flume - flume

I'm getting error when running following command in hadoop
bin/flume-ng agent -c /usr/local/hadoop/flume/conf -f usr/local/hadoop/flume/conf/flume-twitter.conf -n TwitterAgent - flume.root.logger=INFO,console
shows the below error when executing flume command
2016-09-28 17:38:23,508 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: HDFS, type: hdfs
2016-09-28 17:38:23,546 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel MemChannel connected to [Twitter, HDFS]
2016-09-28 17:38:23,565 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{Twitter=EventDrivenSourceRunner: { source:org.apache.flume.source.twitter.TwitterSource{name:Twitter,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#9238ca counterGroup:{ name:null counters:{} } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: MemChannel}} }
2016-09-28 17:38:23,594 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel MemChannel
2016-09-28 17:38:23,653 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: MemChannel: Successfully registered new MBean.
2016-09-28 17:38:23,654 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: MemChannel started
2016-09-28 17:38:23,654 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink HDFS
2016-09-28 17:38:23,654 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source Twitter
2016-09-28 17:38:23,655 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.twitter.TwitterSource.start(TwitterSource.java:131)] Starting twitter source org.apache.flume.source.twitter.TwitterSource{name:Twitter,state:IDLE} ...
2016-09-28 17:38:23,659 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean.
2016-09-28 17:38:23,659 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: HDFS started
2016-09-28 17:38:23,660 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.twitter.TwitterSource.start(TwitterSource.java:139)] Twitter source Twitter started.
2016-09-28 17:38:23,660 (Twitter Stream consumer-1[initializing]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Establishing connection.
2016-09-28 17:38:25,544 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
2016-09-28 17:38:25,545 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Waiting for 10000 milliseconds
2016-09-28 17:38:35,547 (Twitter Stream consumer-1[Waiting for 10000 milliseconds]) [ERROR - org.apache.flume.source.twitter.TwitterSource.onException(TwitterSource.java:331)] Exception while streaming tweets
404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
Relevant discussions can be found on the Internet at:
http://www.google.co.jp/search?q=ec814753 or
http://www.google.co.jp/search?q=0a74cca1
TwitterException{exceptionCode=[ec814753-0a74cca1], statusCode=404, retryAfter=-1, rateLimitStatus=null, featureSpecificRateLimitStatus=null, version=2.2.6}
at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:185)
at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:65)
at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:93)
at twitter4j.TwitterStreamImpl.getSampleStream(TwitterStreamImpl.java:160)
at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:149)
at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:147)
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:426)
2016-09-28 17:38:35,571 (Twitter Stream consumer-1[Waiting for 10000 milliseconds]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Establishing connection.
2016-09-28 17:38:37,049 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
i have added the configuration file it looks like the following
witterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxx
TwitterAgent.sources.Twitter.accessToken = xxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxx
TwitterAgent.sources.Twitter.keywords = INDIA VS NEWZELAND, apache spark, spark, flume, apache mahout, kafka
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:54310/Flume_twitter_data/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
if any one knows please help me
thank u

Related

Boring SSL Handshake Failure and Error copying Identity Cred

I am trying to use Self Signed certificate for my request through Alamofire.
let trustPolicy = ServerTrustPolicy.pinCertificates(certificates:
[certificate], validateCertificateChain: true, validateHost: true)
let serverTrustPolicies: [String: ServerTrustPolicy] = [
"https:-domain-name": trustPolicy,"domain-name" : .disableEvaluation]
let policyManager = ServerTrustPolicyManager(policies: serverTrustPolicies)
But I am getting following error.
CredStore - copyIdentPrefs - Error copying Identity cred.
Error=-25300, query={
class = idnt;
labl = "https://domain-name:443/";
"r_Ref" = 1; }
and
[BoringSSL] boringssl_context_alert_callback_handler(3724)
[C1.1:2][0x139d1bd20] Alert level: fatal, description: handshake
failure 2019-01-22 15:34:23.448605+0530 DB[1276:264543]
[BoringSSL] boringssl_session_errorlog(224) [C1.1:2][0x139d1bd20]
[boringssl_session_handshake_incomplete] SSL_ERROR_SSL(1): operation
failed within the library
2019-01-22 15:34:23.448796+0530
DB[1276:264543] [BoringSSL]
boringssl_session_handshake_error_print(205) [C1.1:2][0x139d1bd20]
5266093016:error:10000410:SSL
routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-109.230.1/ssl/tls_record.cc:586:SSL
alert number 40
2019-01-22 15:34:23.448986+0530 DB[1276:264543]
[BoringSSL] boringssl_context_get_error_code(3545)
[C1.1:2][0x139d1bd20] SSL_AD_HANDSHAKE_FAILURE
2019-01-22
15:34:23.464957+0530 DB[1276:264543] TIC Read Status
[1:0x281599800]: 1:-9824
2019-01-22 15:34:23.467598+0530
DB[1276:264543] Task <43E199F9-B030-4BFD-B9E0-8C9F59B43E72>.<1>
HTTP load failed (error code: -9824 [1:-9824])
2019-01-22
15:34:23.468019+0530 DB[1276:264574] Task
<43E199F9-B030-4BFD-B9E0-8C9F59B43E72>.<1> finished with error - code:
-9824 2019-01-22 15:34:23.470149+0530 DB[1276:264574] Task <43E199F9-B030-4BFD-B9E0-8C9F59B43E72>.<1> load failed with error
Error Domain=NSPOSIXErrorDomain Code=-9824 "Unknown error: -9824"
UserInfo={_NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask
<43E199F9-B030-4BFD-B9E0-8C9F59B43E72>.<1>,
_kCFStreamErrorDomainKey=1, NSErrorPeerAddressKey={length = 16, capacity = 16, bytes =
0x100201bb03106e120000000000000000}, _kCFStreamErrorCodeKey=-9824,
_NSURLErrorRelatedURLSessionTaskErrorKey=(
"LocalDataTask <43E199F9-B030-4BFD-B9E0-8C9F59B43E72>.<1>" )} [-9824]
Have been trying this from past 1 week. Any help would be appreciated.
Disabling evaluation in Alamofire doesn't disable the system's base verification of certificates which blocks self-signed certificates by default. I suggest you read Apple's ATS documentation, but I'm guessing this will require some combination of NSExceptionAllowsInsecureHTTPLoads for your domain and other settings. You should also make sure not to ship those settings if you can help it.

How to configure Corda network using docker (using YAML file)

I'm struggling to configure my Corda network (which is really similar to https://github.com/corda/cordapp-example) using docker, I'm just missing examples with a real network like this, load cordapps is easy (all examples I found so far loads and yo sample like this one: https://github.com/corda/corda-docker/tree/master/plugins), but commit the transaction is something totally different (shouldn't be hard, I agree!).
In order to commit the transaction, consensus in the network and notary approval are needed, and if nodes in the network are not communicating both requirements are not possible.
I'm getting these logs from Xxxxx node when I try to commit a transation to the ledger (As you can see the transaction is mapped to reach localhost:10010 instead of expected xxxxxx:10010):
[WARN ] 2018-09-20T20:47:03,246Z [main] utilities.AppendOnlyPersistentMapBase.set - Double insert in net.corda.node.utilities.AppendOnlyPersistentMap for entity class class net.corda.node.services.identity.PersistentIdentityService$PersistentIdentity key E66540FF121D732F4417B293203D1C61F9F5A467A19AC21EE0327665BA0CA561, not inserting the second time {}
[INFO ] 2018-09-20T20:47:03,257Z [main] messaging.P2PMessagingClient.updateBridgesOnNetworkChange - Updating bridges on network map change: NodeInfo(addresses=[xxxxx:10002], legalIdentitiesAndCerts=[O=Xxxxx, L=New York, C=US], platformVersion=3, serial=1537476408266) {}
[INFO ] 2018-09-20T20:47:03,465Z [main] BasicInfo.printBasicNodeInfo - Loaded CorDapps : example-cordapp-0.1, corda-finance-3.2-corda, corda-core-3.2-corda {}
[INFO ] 2018-09-20T20:47:03,481Z [main] BasicInfo.printBasicNodeInfo - Node for "Xxxxx" started up and registered in 55.5 sec {}
[INFO ] 2018-09-20T20:47:03,486Z [main] messaging.RPCServer.start - Starting RPC server with configuration RPCServerConfiguration(rpcThreadPoolSize=4, reapInterval=PT1S, deduplicationCacheExpiry=PT24H) {}
[INFO ] 2018-09-20T20:47:04,278Z [Thread-0 (ActiveMQ-client-global-threads)] messaging.RPCServer.clientArtemisMessageHandler - SUBMITTING {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=df81b170-c57d-4d2c-ac5e-c50b2dbc951d, invocation_timestamp=2018-09-20T20:47:04.249Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:47:07,901Z [Thread-0 (ActiveMQ-client-global-threads)] messaging.RPCServer.clientArtemisMessageHandler - SUBMITTING {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=abaaafe5-91a9-450f-9b04-078c4446697d, invocation_timestamp=2018-09-20T20:47:07.901Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:47:08,572Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Adding node with info: NodeInfo(addresses=[localhost:10010], legalIdentitiesAndCerts=[O=Xxxxxx, L=New York, C=US], platformVersion=3, serial=1537381493186) {}
[INFO ] 2018-09-20T20:47:08,660Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - No previous node found {}
[INFO ] 2018-09-20T20:47:08,938Z [RxIoScheduler-2] messaging.P2PMessagingClient.updateBridgesOnNetworkChange - Updating bridges on network map change: NodeInfo(addresses=[localhost:10010], legalIdentitiesAndCerts=[O=Xxxxxx, L=New York, C=US], platformVersion=3, serial=1537381493186) {}
[INFO ] 2018-09-20T20:47:09,053Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Done adding node with info: NodeInfo(addresses=[localhost:10010], legalIdentitiesAndCerts=[O=Xxxxxx, L=New York, C=US], platformVersion=3, serial=1537381493186) {}
[INFO ] 2018-09-20T20:47:09,053Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Adding node with info: NodeInfo(addresses=[localhost:10013], legalIdentitiesAndCerts=[O=Xxxxxxx, L=New York, C=US], platformVersion=3, serial=1537381494853) {}
[INFO ] 2018-09-20T20:47:09,056Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - No previous node found {}
[INFO ] 2018-09-20T20:47:09,138Z [RxIoScheduler-2] messaging.P2PMessagingClient.updateBridgesOnNetworkChange - Updating bridges on network map change: NodeInfo(addresses=[localhost:10013], legalIdentitiesAndCerts=[O=Xxxxxxx, L=New York, C=US], platformVersion=3, serial=1537381494853) {}
[INFO ] 2018-09-20T20:47:09,161Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Done adding node with info: NodeInfo(addresses=[localhost:10013], legalIdentitiesAndCerts=[O=Xxxxxxx, L=New York, C=US], platformVersion=3, serial=1537381494853) {}
[INFO ] 2018-09-20T20:47:09,161Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Adding node with info: NodeInfo(addresses=[localhost:10007], legalIdentitiesAndCerts=[O=Xxxxx, L=New York, C=US], platformVersion=3, serial=1537381494948) {}
[INFO ] 2018-09-20T20:47:09,237Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Discarding older nodeInfo for O=Xxxxx, L=New York, C=US {}
[INFO ] 2018-09-20T20:47:09,237Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Adding node with info: NodeInfo(addresses=[localhost:10006], legalIdentitiesAndCerts=[O=Notary, L=New York, C=US], platformVersion=3, serial=1537381495396) {}
[INFO ] 2018-09-20T20:47:09,263Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - No previous node found {}
[INFO ] 2018-09-20T20:47:09,347Z [RxIoScheduler-2] messaging.P2PMessagingClient.updateBridgesOnNetworkChange - Updating bridges on network map change: NodeInfo(addresses=[localhost:10006], legalIdentitiesAndCerts=[O=Notary, L=New York, C=US], platformVersion=3, serial=1537381495396) {}
[INFO ] 2018-09-20T20:47:09,358Z [RxIoScheduler-2] network.PersistentNetworkMapCache.addNode - Done adding node with info: NodeInfo(addresses=[localhost:10006], legalIdentitiesAndCerts=[O=Notary, L=New York, C=US], platformVersion=3, serial=1537381495396) {}
[INFO ] 2018-09-20T20:55:56,465Z [Thread-1 (ActiveMQ-client-global-threads)] messaging.RPCServer.clientArtemisMessageHandler - SUBMITTING {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=bb90b749-06e8-4f8d-9ef1-b841d0e7be8e, invocation_timestamp=2018-09-20T20:55:56.465Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:55:56,529Z [Thread-1 (ActiveMQ-client-global-threads)] messaging.RPCServer.clientArtemisMessageHandler - SUBMITTING {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=3b9a0d1b-309f-4a08-bd6d-2e332ac7e069, invocation_timestamp=2018-09-20T20:55:56.529Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:55:57,251Z [Node thread-1] flow.[8429c030-3a58-4c1a-985c-64eca7f4c54e].initiateSession - Initiating flow session with party O=Xxxxxx, L=New York, C=US. Session id for tracing purposes is SessionId(toLong=8801945676362020052). {}
[INFO ] 2018-09-20T20:55:57,371Z [Messaging DLGWNKZHEid91BXSUY1sSxtGkcoJRjwy3NCrXHXWzsxcNU] messaging.P2PMessagingClient.createQueueIfAbsent - Create fresh queue internal.peers.DL6ZbP6hVmkL3w2rysrMYHchy7axULJssDPkjUzxvn9DB6 bound on same address {}
[INFO ] 2018-09-20T20:55:57,439Z [Thread-1 (ActiveMQ-client-global-threads)] bridging.BridgeControlListener.processControlMessage - Received bridge control message Create(nodeIdentity=DLGWNKZHEid91BXSUY1sSxtGkcoJRjwy3NCrXHXWzsxcNU, bridgeInfo=BridgeEntry(queueName=internal.peers.DL6ZbP6hVmkL3w2rysrMYHchy7axULJssDPkjUzxvn9DB6, targets=[localhost:10010], legalNames=[O=Xxxxxx, L=New York, C=US])) {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=3b9a0d1b-309f-4a08-bd6d-2e332ac7e069, invocation_timestamp=2018-09-20T20:55:56.529Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:55:57,462Z [Thread-1 (ActiveMQ-client-global-threads)] peers.DL6ZbP6hVmkL3w2rysrMYHchy7axULJssDPkjUzxvn9DB6 -> localhost:10010:O=Xxxxxx, L=New York, C=US.start - Create new AMQP bridge {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=3b9a0d1b-309f-4a08-bd6d-2e332ac7e069, invocation_timestamp=2018-09-20T20:55:56.529Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:55:57,474Z [Thread-1 (ActiveMQ-client-global-threads)] netty.AMQPClient.start - connect to: localhost:10010 {actor_id=user1, actor_owningIdentity=O=Xxxxx, L=New York, C=US, actor_store_id=NODE_CONFIG, invocation_id=3b9a0d1b-309f-4a08-bd6d-2e332ac7e069, invocation_timestamp=2018-09-20T20:55:56.529Z, session_id=84058489-40f2-4b91-9527-8e0cfe188294, session_timestamp=2018-09-20T20:46:55.554Z}
[INFO ] 2018-09-20T20:55:57,569Z [nioEventLoopGroup-2-1] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:55:58,571Z [nioEventLoopGroup-2-2] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:55:58,574Z [nioEventLoopGroup-2-3] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:55:59,576Z [nioEventLoopGroup-2-4] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:55:59,580Z [nioEventLoopGroup-2-5] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:00,582Z [nioEventLoopGroup-2-6] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:00,589Z [nioEventLoopGroup-2-7] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:01,591Z [nioEventLoopGroup-2-8] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:01,593Z [nioEventLoopGroup-2-1] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:02,595Z [nioEventLoopGroup-2-2] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:02,599Z [nioEventLoopGroup-2-3] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:03,600Z [nioEventLoopGroup-2-4] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:03,603Z [nioEventLoopGroup-2-5] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:04,604Z [nioEventLoopGroup-2-6] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:04,606Z [nioEventLoopGroup-2-7] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:05,607Z [nioEventLoopGroup-2-8] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:05,610Z [nioEventLoopGroup-2-1] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:06,612Z [nioEventLoopGroup-2-2] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:06,614Z [nioEventLoopGroup-2-3] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:07,616Z [nioEventLoopGroup-2-4] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:07,618Z [nioEventLoopGroup-2-5] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:08,620Z [nioEventLoopGroup-2-6] netty.AMQPClient.run - Retry connect to localhost:10010 {}
[INFO ] 2018-09-20T20:56:08,623Z [nioEventLoopGroup-2-7] netty.AMQPClient.operationComplete - Failed to connect to localhost:10010 {}
This is a node.conf file related to one of my nodes (all the others are in same format):
myLegalName="O=Xxxxx,L=New York,C=US"
basedir : "/opt/corda"
p2pAddress : "xxxxx:10002"
webAddress : "xxxxx:10004"
rpcSettings {
useSsl=false
address="xxxxx:10003"
adminAddress="xxxxx:10051"
}
keyStorePassword : "cordacadevpass"
trustStorePassword : "trustpass"
h2port : 11000
useHTTPS : false
devMode : true
rpcUsers=[
{
password=test
permissions=[
ALL
]
user=user1
}
]
This is the notary node.conf file:
basedir : "/opt/corda"
p2pAddress : "notary:10002"
webAddress : "notary:10004"
h2port : 11000
myLegalName="O=Notary,L=New York,C=US"
detectPublicIp=false
keyStorePassword : "cordacadevpass"
trustStorePassword : "trustpass"
extraAdvertisedServiceIds: [ "corda.notary.simple" ]
useHTTPS : false
devMode : true
rpcSettings = {
useSsl=false
address="notary:10003"
adminAddress="notary:10052"
}
notary {
validating=false
}
rpcUsers=[
{
password=test
permissions=[
ALL
]
user=user1
}
]
This is the docker-compose.yml file:
version: '3.3'
services:
Notary:
networks:
- corda
build:
context: .
args:
BUILDTIME_CORDA_VERSION: 3.2-corda
env_file:
- ./corda_docker.env
ports:
- "10002:10002"
image: corda/notary:2.0
container_name: notary
volumes:
- ./java-source/build/nodes/Notary/node.conf:/opt/corda/node.conf
- ./java-source/build/nodes/Notary/network-parameters:/opt/corda/network-parameters
- ./java-source/build/nodes/Notary/additional-node-infos:/opt/corda/additional-node-infos
- ./java-source/build/nodes/Notary/certificates/:/opt/corda/certificates/
- ./java-source/build/nodes/Notary/cordapps/:/opt/corda/cordapps/
Xxxxx:
networks:
- corda
build:
context: .
args:
BUILDTIME_CORDA_VERSION: 3.2-corda
env_file:
- ./corda_docker.env
ports:
- "10007:10002"
- "10008:10003"
- "10009:10004"
- "10048:10048"
image: corda/xxxxx:2.0
container_name: xxxxx
volumes:
- ./java-source/build/nodes/Xxxxx/node.conf:/opt/corda/node.conf
- ./java-source/build/nodes/Xxxxx/network-parameters:/opt/corda/network-parameters
- ./java-source/build/nodes/Xxxxx/additional-node-infos:/opt/corda/additional-node-infos
- ./java-source/build/nodes/Xxxxx/certificates/:/opt/corda/certificates/
- ./java-source/build/nodes/Xxxxx/cordapps/:/opt/corda/cordapps/
Xxxxxx:
networks:
- corda
build:
context: .
args:
BUILDTIME_CORDA_VERSION: 3.2-corda
env_file:
- ./corda_docker.env
ports:
- "10010:10002"
- "10011:10003"
- "10051:10051"
- "8888:10004"
image: corda/xxxxxx:2.0
container_name: xxxxxx
volumes:
- ./java-source/build/nodes/Xxxxxx/node.conf:/opt/corda/node.conf
- ./java-source/build/nodes/Xxxxxx/network-parameters:/opt/corda/network-parameters
- ./java-source/build/nodes/Xxxxxx/additional-node-infos:/opt/corda/additional-node-infos
- ./java-source/build/nodes/Xxxxxx/certificates/:/opt/corda/certificates/
- ./java-source/build/nodes/Xxxxxx/cordapps/:/opt/corda/cordapps/
Xxxxxxx:
networks:
- corda
build:
context: .
args:
BUILDTIME_CORDA_VERSION: 3.2-corda
env_file:
- ./corda_docker.env
ports:
- "10013:10002"
- "10014:10003"
- "10015:10004"
- "10054:10054"
image: corda/xxxxxxx:2.0
container_name: xxxxxxx
volumes:
- ./java-source/build/nodes/Xxxxxxx/node.conf:/opt/corda/node.conf
- ./java-source/build/nodes/Xxxxxxx/network-parameters:/opt/corda/network-parameters
- ./java-source/build/nodes/Xxxxxxx/additional-node-infos:/opt/corda/additional-node-infos
- ./java-source/build/nodes/Xxxxxxx/certificates/:/opt/corda/certificates/
- ./java-source/build/nodes/Xxxxxxx/cordapps/:/opt/corda/cordapps/
networks:
corda:
Which changes should I do in order to map correctly the nodes? As you can see, the node is calling itself instead of calling the other peer.
This transaction involves two peers.
It looks like you tried create your nodes by plugin: net.corda.plugins.Cordform.
But for nodes for Docker you should use net.corda.plugins.Dockerform.
Example for prepare Docker Nodes

Apache Camel sending multiple REST requests without timeout

i'm trying to send many REST requests via camel. I expected a sequential
processing but after 11 requests the test stopped and it started producing timeouts.
Is it really necessary to configure the restlet thread pool settings described here? Why is the processing stopping after some requests? I don't want parallelism...
Thank you in advance
#RunWith(CamelSpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {MultipleRestRequestsTest.ContextConfig.class}, loader = CamelSpringDelegatingTestContextLoader.class)
public class MultipleRestRequestsTest extends AbstractJUnit4SpringContextTests {
#EndpointInject(uri = "mock:result")
protected MockEndpoint resultEndpoint;
#Produce(uri = "direct:start")
protected ProducerTemplate template;
#Test
public void testMultipleRequests() throws Exception {
for (int i = 1; i <= 1000; i++) {
template.sendBody(i);
}
resultEndpoint.setExpectedCount(1000);
resultEndpoint.assertIsNotSatisfied(50000);
}
#Configuration
public static class ContextConfig extends SingleRouteCamelConfiguration {
#Bean(name={"restlet"})
public RestletComponent restlet() {
RestletComponent restlet = new RestletComponent();
restlet.setMaxThreads(100);
restlet.setThreadMaxIdleTimeMs(10000);
restlet.setMaxQueued(20);
return restlet;
}
#Bean
public RouteBuilder route() {
return new RouteBuilder() {
public void configure() {
from("direct:start")
.log("Request ${body}")
.to("restlet://http://localhost:8080/mock-service")
.to("mock:result");
}
};
}
}
}
INFORMATION: Starting the Apache HTTP client 2017-06-07 08:06:27,761
INFO SpringCamelContext:2835 - Apache Camel 2.17.0 (CamelContext:
camel-1) started in 0.440 seconds 2017-06-07 08:06:27,784
INFO route1:159 - Request 1 2017-06-07 08:06:27,926 INFO
route1:159 - Request 2 2017-06-07 08:06:27,931 INFO
route1:159 - Request 3 2017-06-07 08:06:27,935 INFO
route1:159 - Request 4 2017-06-07 08:06:27,940 INFO
route1:159 - Request 5 2017-06-07 08:06:27,944 INFO
route1:159 - Request 6 2017-06-07 08:06:27,949 INFO
route1:159 - Request 7 2017-06-07 08:06:27,953 INFO
route1:159 - Request 8 2017-06-07 08:06:27,958 INFO
route1:159 - Request 9 2017-06-07 08:06:27,962 INFO
route1:159 - Request 10 2017-06-07 08:06:27,967 INFO
route1:159 - Request 11 Jun 07, 2017 8:06:57 AM
org.restlet.ext.httpclient.internal.HttpMethodCall sendRequest
WARNUNG: An error occurred during the communication with the remote
HTTP server. org.apache.http.conn.ConnectionPoolTimeoutException:
Timeout waiting for connection from pool at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:412)
at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298)
at
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at
org.restlet.ext.httpclient.internal.HttpMethodCall.sendRequest(HttpMethodCall.java:339)
at
org.restlet.ext.httpclient.internal.HttpMethodCall.sendRequest(HttpMethodCall.java:363)
at
org.restlet.engine.adapter.ClientAdapter.commit(ClientAdapter.java:81)
at
org.restlet.engine.adapter.HttpClientHelper.handle(HttpClientHelper.java:119)
at org.restlet.Client.handle(Client.java:153) at
org.restlet.Restlet.handle(Restlet.java:342) at
org.restlet.Restlet.handle(Restlet.java:355)

PHPMailer dev-xoauth failed authentication

So I've gone through all of the docs on github - phpmailer xoauth tree to set up my scripts accordingly to use Google's oauth2. I am trying to send an email from the form at halcyonco.io to myself. The domain name is just to live test this site. I encounter the same error when sending the form locally.
My question is when I submit the form I receive this error:
SERVER -> CLIENT: 220 mx.google.com ESMTP pj4sm27148656pbb.29 - gsmtp
CLIENT -> SERVER: EHLO halcyonco.io
SERVER -> CLIENT: 250-mx.google.com at your service, [68.65.121.206]250-SIZE 35882577250-8BITMIME250-STARTTLS250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING250 SMTPUTF8
CLIENT -> SERVER: STARTTLS
SERVER -> CLIENT: 220 2.0.0 Ready to start TLS
CLIENT -> SERVER: EHLO halcyonco.io
SERVER -> CLIENT: 250-mx.google.com at your service, [68.65.121.206]
250-SIZE 35882577250-8BITMIME
250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN XOAUTH
250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING
250 SMTPUTF8
SMTP Error: Could not authenticate.
CLIENT -> SERVER: QUIT
SERVER -> CLIENT: 221 2.0.0 closing connection pj4sm27148656pbb.29 - gsmtp
SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
Mailer Error: SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
I have been in touch with Namecheap as I have my hosting and domains through them but after an hour and half they could not help me.
Below is my contact.php script
<?php
if ($_SERVER["REQUEST_METHOD"] == "POST") {
$name = trim($_POST["user_name"]);
$email = trim($_POST["user_email"]);
$comment = trim($_POST["comment"]);
if ($name == "" OR $email == "" OR $comment == "") {
echo '<h3>Please fill out all forms.</h3>';
exit;
}
// SPAM protection
foreach ($_POST as $value) {
if ( stripos($value, 'Content-Type') !== FALSE ) {
echo "There was a problem with the information you entered.";
exit;
}
}
//SMTP needs accurate times, and the PHP time zone MUST be set
//This should be done in your php.ini, but this is how to do it if you don't have access to that
date_default_timezone_set('America/New_York');
require '../vendor/phpmailer/phpmailer/PHPMailerAutoload.php';
//Load dependnecies from composer
//If this causes an error, run 'composer install'
require '../vendor/autoload.php';
//Create a new PHPMailer instance
$mail = new PHPMailerOAuth;
//Tell PHPMailer to use SMTP
$mail->isSMTP();
//Enable SMTP debugging
// 0 = off (for production use)
// 1 = client messages
// 2 = client and server messages
$mail->SMTPDebug = 2;
//Ask for HTML-friendly debug output
$mail->Debugoutput = 'html';
//Set the hostname of the mail server
$mail->Host = 'smtp.gmail.com';
//Set the SMTP port number - 587 for authenticated TLS, a.k.a. RFC4409 SMTP submission
$mail->Port = 587;
//Set the encryption system to use - ssl (deprecated) or tls
$mail->SMTPSecure = 'tls';
//Whether to use SMTP authentication
$mail->SMTPAuth = true;
//Set AuthType
$mail->AuthType = 'XOAUTH2';
//User Email to use for SMTP authentication - Use the same Email used in Google Developer Console
$mail->oauthUserEmail = "myemail#gmail.com";
//Obtained From Google Developer Console
$mail->oauthClientId = "1234567890.apps.googleusercontent.com";
//Obtained From Google Developer Console
$mail->oauthClientSecret = "123456789";
//Obtained By running get_oauth_token.php after setting up APP in Google Developer Console.
//Set Redirect URI in Developer Console as [https/http]://<yourdomain>/<folder>/get_oauth_token.php
// eg: http://localhost/phpmail/get_oauth_token.php
$mail->oauthRefreshToken = "1/ABCD12345EFGH6789";
//Set who the message is to be sent from
//For gmail, this generally needs to be the same as the user you logged in as
$mail->setFrom($email, $name);
//Set who the message is to be sent to
$mail->addAddress('myemail#gmail.com', 'Brandon Smith');
//Set the subject line
$mail->Subject = 'Promethean Fitness Enquiry | ' . $name;
//Read an HTML message body from an external file, convert referenced images to embedded,
//convert HTML into a basic plain-text alternative body
$mail->msgHTML = $comment;
//Replace the plain text body with one created manually
$mail->Body = $comment;
$mail->AltBody = $comment;
//Attach an image file
//send the message, check for errors
if (!$mail->send()) {
echo '<h3>Please ensure you have entered a correct email address.</h3><br>';
echo "Mailer Error: " . $mail->ErrorInfo;
} else {
header("Location: ../html/thanks.html");
exit;
}
//This is a spam protection
if ($_POST["user_address"] != "") {
echo "Your form submission has an error";
exit;
}
}
?>
Any help on what I need to do in order to fix this would be great.

AWS S3 File Download with Objective-C Successful Then Unsuccessful

I have a strange problem using AWS S3. The first time I call the code, files are downloaded from the S3 service as expected. The next time I call the code to download additional files, it hangs.
I am using version 2.1.2 of the AWS iOS SDK. I have downloaded and installed the SDK using cocoapods. I have read other threads on Stack Overflow and they do not address this issue.
The code:
static AWSStaticCredentialsProvider *credentialsProvider;
static AWSServiceConfiguration *configuration;
static AWSS3 *transferManager;
//This code is run when the object is instantiated
-(id)init
{
if ( self = [super init] ) {
credentialsProvider = [AWSStaticCredentialsProvider credentialsWithAccessKey:#“ACCESS KEY" secretKey:#“SECRET"];
configuration = [AWSServiceConfiguration configurationWithRegion:AWSRegionUSEast1 credentialsProvider:credentialsProvider];
[AWSServiceManager defaultServiceManager].defaultServiceConfiguration = configuration;
transferManager = [[AWSS3 alloc] initWithConfiguration:configuration];
}
return self;
}
-(UIImage *) getImageData:(NSString *)imageName
{
AWSS3 *transferManager = [[AWSS3 alloc] initWithConfiguration:configuration];
AWSS3GetObjectRequest *getImageRequest = [AWSS3GetObjectRequest new];
getImageRequest.bucket = #"quizontapimages/quizimages";
getImageRequest.key = imageName;
getImageRequest.key = [getImageRequest.key stringByAppendingString:#".png"];
BFTask *downloadTask1 = [[transferManager getObject:getImageRequest] continueWithExecutor:[BFExecutor mainThreadExecutor] withBlock:^id(BFTask *task) {
if(task.error)
{
NSLog(#"Error: %#",task.error);
}
else
{
NSData *data = [task.result body];
image = [UIImage imageWithData:data];
}
return nil;
}];
[downloadTask1 waitUntilFinished];
return image;
}
I have also tried commenting out the first line of getImageData method to use the initialized transferManager with the same results.
When I use AWS verbose logging on the successful download I get the following information:
2015-06-11 16:12:51.199 QuizOnTap[7362:538728] AWSiOSSDKv2 [Debug] AWSSignature.m line:241 | -[AWSSignatureV4Signer signS3RequestV4:] | Canonical request: [GET
/quizontapimages/quizimages/WABD_QSI_333x118.png
content-type:binary/octet-stream
host:s3.amazonaws.com
user-agent:aws-sdk-iOS/2.0.17 iPhone-OS/8.3 en_US
x-amz-content-sha256:*some key*
x-amz-date:20150611T211251Z
content-type;host;user-agent;x-amz-content-sha256;x-amz-date
*some key*]
2015-06-11 16:12:51.199 QuizOnTap[7362:538728] AWSiOSSDKv2 [Debug] AWSSignature.m line:248 | -[AWSSignatureV4Signer signS3RequestV4:] | AWS4 String to Sign: [AWS4-HMAC-SHA256
20150611T211251Z
20150611/us-east-1/s3/aws4_request
*some key*]
2015-06-11 16:12:51.493 QuizOnTap[7362:539182] AWSiOSSDKv2 [Debug] AWSURLResponseSerialization.m line:258 | -[AWSXMLResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response header: [{
"Accept-Ranges" = bytes;
"Content-Length" = 15947;
"Content-Type" = "image/png";
Date = "Thu, 11 Jun 2015 21:12:52 GMT";
Etag = "\"ff690cf7807b538278fc1590ce446785\"";
"Last-Modified" = "Sun, 31 May 2015 16:14:42 GMT";
Server = AmazonS3;
"x-amz-id-2" = "QIu6OWhmuO53z6Qgo+Q/4gsN4dQbyCAPbwS4QyDm/pmmSRXj8M5O4x5GoXMQq/rgSh0AKqt0uVk=";
"x-amz-request-id" = E0DB22A10F901130;
}]
2015-06-11 16:12:51.493 QuizOnTap[7362:539182] AWSiOSSDKv2 [Verbose] AWSURLResponseSerialization.m line:263 | -[AWSXMLResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response body: [(null)]
Later on in the process I call the same identical code that never returns from [downloadTask1 waitUntilFinished];. Below is the AWS verbose log for that transaction. Notice the long running operation warning as well.
2015-06-11 16:14:03.075 QuizOnTap[7362:538689] AWSiOSSDKv2 [Debug] AWSSignature.m line:241 | -[AWSSignatureV4Signer signS3RequestV4:] | Canonical request: [GET
/quizontapimages/quizimages/neat_square_1.png
content-type:binary/octet-stream
host:s3.amazonaws.com
user-agent:aws-sdk-iOS/2.0.17 iPhone-OS/8.3 en_US
x-amz-content-sha256:*some key*
x-amz-date:20150611T211403Z
content-type;host;user-agent;x-amz-content-sha256;x-amz-date
*some key*]
2015-06-11 16:14:03.075 QuizOnTap[7362:538689] AWSiOSSDKv2 [Debug] AWSSignature.m line:248 | -[AWSSignatureV4Signer signS3RequestV4:] | AWS4 String to Sign: [AWS4-HMAC-SHA256
20150611T211403Z
20150611/us-east-1/s3/aws4_request
*some key*]
2015-06-11 16:14:03.076 QuizOnTap[7362:538689] Warning: A long-running operation is being executed on the main thread.
Break on warnBlockingOperationOnMainThread() to debug.
2015-06-11 16:14:03.396 QuizOnTap[7362:540085] AWSiOSSDKv2 [Debug] AWSURLResponseSerialization.m line:258 | -[AWSXMLResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response header: [{
"Accept-Ranges" = bytes;
"Content-Length" = 4998;
"Content-Type" = "image/png";
Date = "Thu, 11 Jun 2015 21:14:04 GMT";
Etag = "\"7b27202a6560ed99277acebf2235ba89\"";
"Last-Modified" = "Fri, 22 May 2015 15:16:13 GMT";
Server = AmazonS3;
"x-amz-id-2" = "5TxaPQy8jblFr8qVXfiSg3pK0EfOqSdEWdykE0kfRKUdrfxvmxvYvzf0uXGYiXXliAG/DbM55tM=";
"x-amz-request-id" = 9F7BC6AC5494D285;
}]
2015-06-11 16:14:03.396 QuizOnTap[7362:540085] AWSiOSSDKv2 [Verbose] AWSURLResponseSerialization.m line:263 | -[AWSXMLResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response body: [(null)]
You are not retaining a strong reference to the transferManager object in - getImageData:. Please remember that - getObject: is an asynchronous method, and it returns immediately. You need to retain a strong reference to the service client until the request finishes processing.
If you use the AWS Mobile SDK for iOS 2.1.2, Xcode should give you a compiler warning for the use of - initWithConfiguration:. The method was deprecated to mitigate the misuse of the API such as this case. Please use + defaultS3TransferManager or + S3TransferManagerForKey: to retrieve the AWSS3TransferManager object.
(Also, the log indicates you are using 2.0.17 instead of 2.1.2.)

Resources