I have just started looking into flume for writing messages to hdfs using the hdfs sink. I am wondering if flume source can act as a jms consumer for my message broker.
Does flume provide integration with message broker.
Or do i need to write a custom jms client that will push messages to a flume source.
Flume 1.3 does not provide JMS source out of the box. However, it seems that such component will be shipped with the next version, Flume 1.4: https://issues.apache.org/jira/browse/FLUME-924.
Meanwhile you can get the source code of this new component here and build it. AFAIK, Flume 1.4 does not break its interfaces, so probably the component will work with 1.3 as well.
Related
I'm comparing Apache StreamPipes and SCDF (Spring Cloud Dataflow).
I found out that there are some similarities:
Components of the Stream are executed as microservices via Wrappers (Flink/standalone)
Internally uses Message Broker to automatically create required topics and connect pipeline-components by that
I found nothing about a Support for using Kubernetes as Execution Engine. Is something planned in the Future? Anyone knows some other differences/similarities?
I am trying to understand Flume and referring to the official page of flume at flume.apache.org
In particular, referring to this section, I am bit confused in this.
Do we need to run the flume agent on the actual webserver or can we run flume agents in a different physical server and acquire data from webserver?
If above is correct, then how flume agent gets the data from webserver logs? How can webserver make its data available to the flume agent ?
Can anyone help understand this?
The Flume agent must pull data from a source, publish to a channel, which then writes to a sink.
You can install Flume agent in either a local or remote configuration. But, keep in mind that having it remote will add some network latency to your event processing, if you are concerned about that. You can also "multiplex" Flume agents to have one remote aggregation agent, then individual local agents on each web server.
Assuming a flume agent is locally installed using a Spooldir or exec source, it'll essentially tail any file or run that command locally. This is how it would get data from logs.
If the Flume agent is setup as a Syslog or TCP source (see Data ingestion section on network sources), then it can be on a remote machine, and you must establish a network socket in your logging application to publish messages to the other server. This is a similar pattern to Apache Kafka.
In our project we have websphere liberty server and IBM MQ as message server. We are moving in cloud. We want to replace IBM MQ with AWS SQS. In server.xml we have following entries to get IBM MQ connection factory using jndi.
<jmsQueueConnectionFactory connectionManagerRef="ABCConnMgr" jndiName="jms/ABC_QCF">
<properties.wmqJms channel="CH.ABC.SVRCONN" hostName="abc-mq1-st4.ebiz.abc.com" port="21414" queueManager="ABC401" transportType="CLIENT"/>
</jmsQueueConnectionFactory>
<connectionManager agedTimeout="-1s" connectionTimeout="180s" id="ABCConnMgr" maxIdleTime="1800s" maxPoolSize="50" minPoolSize="0" purgePolicy="EntirePool" purgeagedTimeout="-1s" reapTime="180s"/>
<jmsQueue id="ABC.ORDERMGT.INPUT.QA" jndiName="jms/ABC_ORDERMGT_INPUT">
<properties.wmqJms baseQueueName="ABC.ORDERMGT.INPUT.QA"/>
</jmsQueue>
Similarly I am looking jndi configuration for Amazon SQS connection factory and sqs queue. I think IBM need to develop resource adapter for SQS and provide this configuration.
You are correct, you will need a resource adapter for Amazon SQS. It could be provided by any third-party vendor, does not need to be IBM for it to work because JCA is a standard.
Once you have a resource adapter, here are some helpful Knowledge Center links to get it configured in Liberty,
Overview of JCA/JMS config
Configuring Resource Adapters
Configuring JMS Connection Factories
Configuring JMS Destinations
I am able to configure an agent for window but i have a confusion regarding connectivity between web servers logs with agent.
1: How to connect web server with agent ?
2: while starting flume.bat file. It is generating flume.log file in which i am getting below mentioned Exception.
org.apache.flume.conf.ConfigurationException: No channel configured for sink: hdfssink
at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:51)
atorg.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:661)
1.The data flow is as below
your application (or web server) --> source --> channel --> sink
Now, the data can flow from your webserver to the source either by "pull" mechanism or "push" mechanism. In your case, you can either tail the webserver logs or use a spooling source.
2.This looks like a misconfiguration issue. You need to post your config file to figure out the issue
The Grails RabbitMQ plugin is great, but I can't seem to find any information about what to do when processing a message fails. I would prefer not to send an Ack back to RabbitMQ, but I can't seem to find any information on how to do this using the plugin.
Is it possible to control this with the RabbitMQ plugin for grails, or am I stuck not using the plugin?
The current version of Grails RabbitMQ plugin (0.3.3) do not support manual acknowledge mode.
Under the hood Grails RabbitMQ plugin use Spring AMQP libraries, so you can always declare your custom SimpleMessageListenerContainer in resources.groovy and implement a listener with manual acks. Check out http://static.springsource.org/spring-amqp/reference/html/#d0e377