I am trying to understand Flume and referring to the official page of flume at flume.apache.org
In particular, referring to this section, I am bit confused in this.
Do we need to run the flume agent on the actual webserver or can we run flume agents in a different physical server and acquire data from webserver?
If above is correct, then how flume agent gets the data from webserver logs? How can webserver make its data available to the flume agent ?
Can anyone help understand this?
The Flume agent must pull data from a source, publish to a channel, which then writes to a sink.
You can install Flume agent in either a local or remote configuration. But, keep in mind that having it remote will add some network latency to your event processing, if you are concerned about that. You can also "multiplex" Flume agents to have one remote aggregation agent, then individual local agents on each web server.
Assuming a flume agent is locally installed using a Spooldir or exec source, it'll essentially tail any file or run that command locally. This is how it would get data from logs.
If the Flume agent is setup as a Syslog or TCP source (see Data ingestion section on network sources), then it can be on a remote machine, and you must establish a network socket in your logging application to publish messages to the other server. This is a similar pattern to Apache Kafka.
Related
I have a prometheus server running on a K8s instance and telegraf on a different cluster. Is there some way to pull metrics from the prometheus server using telegraf? I know there is telegraf support for scraping metrics from prometheus clients but I am looking to get these metrics from the prometheus server.
Thanks
there is this thing inside data sources, called scrapers, its a tab, you just need to put the url of the server.
I am trying to configure this using cli, but i can only do it with gui
There is a prometheus remote write parser (https://github.com/influxdata/telegraf/tree/master/plugins/parsers/prometheusremotewrite), I think it will be included in the 1.19.0 release of Telegraf. If you want to try it out now you can use a nightly build. (https://github.com/influxdata/telegraf#nightly-builds)
Configure your prometheus remote write towards telegraf and configure the input plugin to listen for traffic on the port that you configured. For convenience sake, have the output plugin configured to file so you can see the metrics in a file, almost immediately
I'd like to keep track of data that might be stuck in Apache Artemis queues and I'd like to leverage its JMX management abilities together with our Zabbix instance.
What steps do I need to take in order to successfully connect Zabbix to Artemis via JMX? The ones mentioned in https://activemq.apache.org/artemis/docs/latest/management.html are not quite clear to me.
I had to disable the internal connector and go the other way around by adding this to the artemis.profile file:
JAVA_ARGS="$JAVA_ARGS -Dcom.sun.management.jmxremote"
JAVA_ARGS="$JAVA_ARGS -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_ARGS="$JAVA_ARGS -Dcom.sun.management.jmxremote.ssl=false"
JAVA_ARGS="$JAVA_ARGS -Dcom.sun.management.jmxremote.port=1099"
JAVA_ARGS="$JAVA_ARGS -Dcom.sun.management.jmxremote.rmi.port=1098"
JAVA_ARGS="$JAVA_ARGS -Djava.rmi.server.hostname=edimq-broker-master-az1.dc01.clouedi.local"
However, this way it's anything but secure, I know.
As the documentation states, you need to add this to your management.xml:
<connector connector-port="1099"/>
This will expose a JMX connector on localhost so if you want to be able to access it remotely from another machine on your network (i.e. your Zabbix instance) then you should do something like:
<connector connector-port="1099" connector-host="myhost" />
Also, if you have multiple IP addresses on the machine hosting the broker you'll want to set this system property in the JAVA_ARGS variable in artemis.profile:
-Djava.rmi.server.hostname=myhost
Then point your Zabbix instance at the broker using a url like:
service:jmx:rmi:///jndi/rmi://myhost:1099/jmxrmi
You can see this in action by running the jmx example shipped with Artemis in the examples/features/standard/ directory. Just navigate into that directory and run mvn verify. Running the example will create a broker instance, start the broker instance, and run the client all automatically. After the example runs you can go to into the target/server0 directory and look at all the configuration files to compare them to your own. You can also start broker independently of the example if you wish (by running ./artemis run from the target/server0/bin directory). Once the broker is running you should be able to connect to it with JConsole no problem using a JMX url like this:
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
For my upcoming bachelor's thesis I want to develop a tool that collects system and application data from Apache Flink and sends this data in some kind of "events" to another system. This tool will be installed on Flink job- and taskmanager nodes. Beside data from linux system utilities like dstat I would like to collect JMX data.
My problem is, that I couldn't figure out how to connect via remote JMX connection by using a port to Flinks jobmanager. Although the collector will be on the same machine, I really try to avoid using a --javaagent to access JMX data of Flink's JVM.
Another problem is, I have a local docker setup based on https://github.com/apache/flink/tree/master/flink-contrib/docker-flink and updated to flink-1.0.2, that I cannot connect via jconsole because I don't know how to "open" a JMX remote port for the job- and taskmanager.
Is there any way to achieve this?
Thanks in advance, any ideas very appreciated.
Solved!
I needed to add env.java.opts: -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
in flink-conf.yaml.
No it's possible to connect the jobmanager via jconsole.
I am able to configure an agent for window but i have a confusion regarding connectivity between web servers logs with agent.
1: How to connect web server with agent ?
2: while starting flume.bat file. It is generating flume.log file in which i am getting below mentioned Exception.
org.apache.flume.conf.ConfigurationException: No channel configured for sink: hdfssink
at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:51)
atorg.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:661)
1.The data flow is as below
your application (or web server) --> source --> channel --> sink
Now, the data can flow from your webserver to the source either by "pull" mechanism or "push" mechanism. In your case, you can either tail the webserver logs or use a spooling source.
2.This looks like a misconfiguration issue. You need to post your config file to figure out the issue
I have a java process on a linux server, which runs with this option: -Dcom.sun.management.jmxremote
So I cannot just connect to this process via jconsole running on my local pc (because neither port nor -Dcom.sun.management.jmxremote.ssl=false options are set up).
But still, how can I connect to the application and run some operations over some of its MBeans? It this possible? I have a ssh access to the server and would be able to run it "locally" on the server (but not changing the options unfortunately)
According to JMX documentation the -Dcom.sun.management.jmxremote option
Enables the JMX remote agent and local monitoring via JMX connector published on a private
interface used by jconsole. The jconsole tool can use this connector if it is executed by
the same user ID as the user ID that started the agent. No password or access files are
checked for requests coming via this connector.
The naming is a bit unfortunate because it in fact enables the local monitoring only.
Since you can not change the options but can access the server via SSH the only option is to use X server forwarding (ssh -X ...) and run jconsole (or better yet jvisualvm which has specific optimisations for running remotely).