I have set up JMeter in Non-GUI mode in docker with influxdb and Grafana.
Jmeter real time results are first shipped in influxdb and shows on Grafana dashboard. Below is the table format which is being shipped in influxdb:
My team requirement is to see the view result tree result in Grafana as they want to see request/response for each transaction as per below:
Please help me to ship the result in influxdb of same as we team want to see each request/response in Grafana otherwise no use of dockerize the Jmeter. I am already using influxdbbackendlistener but unable to get tree result in influxdb.
Related
I am looking at how to get the information on the number of consumers from a Kafka server running in a docker container.
But I'll also take almost any info to help point me in a direction that is forward movement. I've been trying through Python ond URI requests, but I'm getting the feeling I need to get back to Java to ask Kafka questions on it's status?
In relation to the current threads I've seen, many handy scripts from $KAFKA_HOME are referenced but, the configured systems I have access to do not have $KAFKA_HOME defined - nor do they have the contents of that directory. My world is a docker container without a CLI access. So I haven't been able to apply the solutions requiring shell scripts or other tools from $KAFKA_HOME to my running system.
One of the things I have tried is a python script using requests.get(uri...)
where the uri looks like:
http://localhost:9092/connectors/
The code looks like:
r = requests.get("http://%s:%s/connectors" % (config.parameters['kafkaServerIPAddress'],config.parameters['kafkaServerPort']))
currentConnectors=r.json()
So far I get a "nobody's home at that address" response.
I'm really stuck and a pointer to something akin to "Beginners Guide to Getting KAFKA Monitoring Information" would be great. Also if there's a way to grab the helpful kafka shell scripts & tools, that would be great to - where do they come from?
One last thing - I'm new enough to Kafka that I don't know what I don't know.
Thanks.
running in a Docker container
That shouldn't matter, but Confluent maintains a few pages that go over how to configure the containers for monitoring
https://docs.confluent.io/platform/current/installation/docker/operations/monitoring.html
https://docs.confluent.io/platform/current/kafka/monitoring.html
number of consumers
Such a metric doesn't exist
Python and URI requests
You appear to be using the /connectors endpoint of the Kafka Connect REST API (which runs on port 8083, not 9092). It is not a monitoring endpoint for brokers or non-Connect-API consumers
way to grab the helpful kafka shell scripts & tools
https://kafka.apache.org/downloads > Binary downloads
You don't need container shell access, but you will need external network access, just as all clients outside of a container would.
Im new here, I have an issue with one plugin,
So im using telegraf to get data from ipmi sensor, and it’s working I can see that in Grafana dashboard via InfluxDB,
so until here everything is working correctly,
I added another input plugin to my telegraf.conf : telegraf-speedtest/speedtest.conf at master · Derek-K/telegraf-speedtest · GitHub
Once im checking telegraf -test , I can see that the first plugin(ipmi sensor) is OK and second plugin (speedtest) is OK as well.
But speedtest measurements are not stored in the influxdb I check it using
root#d5c51db15460:/# influx -execute ‘show measurements’ -database ‘telegraph’
name: measurements
name
ipmi_sensor
Here as you can see there is only the ipmi_sensor :( .
Telegraf is restarted already, and both plugins are working with -test
im not sure where is the issue, I appreciate your help guys
thank you
The telegraf --test option does not send data to outputs. I would suggest using the --once option if you want to ensure data is making it to outputs as well.
In Jenkins, I want to get information like how many times builds failed in a given period, which tests failed multiple times in successive builds, did each of these failed tests fail due to same or different reasons each time, is a test failing in multiple environments or only some environments etc.
How do I get such information from Jenkins ?
Your question is a bit vague. So I will give you the solution I used to solve this problem with the use of jenkins's influxDB plugin with InfluxDB as a database and Grafana as a Dashboard tool.
Setup InfluxDB
I use the docker image: influxdb:1.7-alpine
mounted volumes /docker-entrypoint-initdb.d and /var/lib/influxdb
In folder /docker-entrypoint-initdb.d I added a file db.iql to create my database
CREATE DATABASE "jenkins" WITH DURATION 24w REPLICATION 1 SHARD DURATION 1d NAME "jenkins_retention_6month"
Setup the InfluxDB plugin
See section configuration of the plugin's page
https://wiki.jenkins.io/display/JENKINS/InfluxDB+Plugin
Use the plugin
the InfluxDbPublisher step can be used to collect data using plugins like the Metrics Plugin, however I use it with customDataMap
influxDbPublisher(
selectedTarget: 'myTarget',
customDataMap: [
myMeasure: [
field: value
]
],
customDataMapTags: [
myMeasure: [
tag: 'someTag'
]
]
])
Everything is documented on
https://wiki.jenkins.io/display/JENKINS/InfluxDB+Plugin
Setup Grafana
I use the docker image: grafana/grafana:6.4.3
I mounted volume /var/lib/grafana
When the instance of grafana is running, add your influxdb database as a datasource
I configured grafana with the following environment variables:
GF_SERVER_DOMAIN=grafana.mydomain.com
GF_SECURITY_ADMIN_PASSWORD=MyPassword
GF_SMTP_ENABLED=true
GF_SMTP_HOST=smtp:25
GF_SMTP_FROM_ADDRESS=grafana#grafana.mydomain.com
I used docker image namshi/smtp to get a smtp server
Create Grafana Dashboards
It is very easy to create a new dashboard with the auto completion feature of grafana. You will certainly need to tweak few times the data you sent with the influxDbPublisher step.
Now you have your dashboards, you can setup alerts in order to get notified in advance by email when something od is happening with your CI
Here is the scenario I want to resolve: I have two environments: a local machine and a virtual machine hosted in Azure
In the virtual machine I start a gremlin container which includes the gremlin client, server and connects to a cassandra graph database.
This is the information of the container running when i run the docker container ls command:
CONTAINER ID: 029095e26f53
IMAGE: 3f03c6bfb0a2
COMMAND: "/bin/sh -c /gremlin…"
CREATED: 2 weeks ago
STATUS: Up 2 weeks
PORTS: 0.0.0.0:8182->8182/tcp
NAME: gremlin
When I enter inside the container, I run the following command in order to run the gremlin client:
./bin/gremlin.sh
Once inside the gremlin console i run the following command to connect to the tinkerpop server:
:remote connect tinkerpop.server conf/remote.yaml
==>Connected - localhost/127.0.0.1:8182 ---> answer from gremlin console
If I run the following gremlin query:
:> g.V().count()
I get a number different from zero, telling me that there are a records on the graph database.
Now on the other side I have the Gephi client on my local machine which I want it to be able to show that graph database. Or at least, make Gephi to show the the visual data from a
graph = TinkerFactory.createModern()
running inside the gremlin container.
I want to do this because I need to choose a visualization tool for gremlin and titan ecosystem.
I tried to set up Gephi client feature to connect to the virtual machine's ip and the port 8182 but it shows me the red dot telling me that is not possible. What am i missing? I am pretty sure there are a few steps missing. Thanks in advance,
Juan Ignacio
If your graph is "remote" and not in-memory in the Gremlin Console then you have to devise a way to make it available locally that way. This situation is typical for Graphs that run in Gremlin Server or are wholly remote like CosmosDB, DSE Graph or Amazon Neptune.
They typical method to make it available locally is to use [subgraph()][1]-step to pull out just the portion of the graph that you care about and return that to the Gremlin Console. It will be returned as a TinkerGraph for graphs that support subgraph()-step (like Titan, though I assume you would use JanusGraph), so for your test which is using TinkerFactory and a tiny graph you could just do this:
gremlin> :remote connect tinkerpop.server conf/remote-objects.yaml
Note the configuration of "remote-objects.yaml" because that configuration will return actual objects - an actual TinkerGraph rather than a string representation of a TinkerGraph.
gremlin> :> TinkerFactory.createModern()
That will create the "modern" graph remotely and return the TinkerGraph to the Gremlin Console. You can access that result
gremlin> graph = result[0].object
The :> stores the response from the server in a variable named "result" and that contains your TinkerGraph in a List. This is explained in the reference documentation. From there you can use that "graph" object as you would using the standard Gephi instructions.
I am currently trying to break into Data engineering and I figured the best way to do this was to get a basic understanding of the Hadoop stack(played around with Cloudera quickstart VM/went through tutorial) and then try to build my own project. I want to build a data pipeline that ingests twitter data, store it in HDFS or HBASE, and then run some sort of analytics on the stored data. I would also prefer that I use real time streaming data, not historical/batch data. My data flow would look like this:
Twitter Stream API --> Flume --> HDFS --> Spark/MapReduce --> Some DB
Does this look like a good way to bring in my data and analyze it?
Also, how would you guys recommend I host/store all this?
Would it be better to have one instance on AWS ec2 for hadoop to run on? or should I run it all in a local vm on my desktop?
I plan to have only one node cluster to start.
First of all, Spark Streaming can read from Twitter, and in CDH, I believe that is the streaming framework of choice.
Your pipeline is reasonable, though I might suggest using Apache NiFi (which is in the Hortonworks HDF distribution), or Streamsets, which is installable in CDH easily, from what I understand.
Note, these are running completely independently of Hadoop. Hint: Docker works great with them. HDFS and YARN are really the only complex components that I would rely on a pre-configured VM for.
Both Nifi and Streamsets give you a drop and drop UI for hooking Twitter to HDFS and "other DB".
Flume can work, and one pipeline is easy, but it just hasn't matured at the level of the other streaming platforms. Personally, I like a Logstash -> Kafka -> Spark Streaming pipeline better, for example because Logstash configuration files are nicer to work with (Twitter plugin builtin). And Kafka works with a bunch of tools.
You could also try out Kafka with Kafka Connect, or use Apache Flink for the whole pipeline.
Primary takeaway, you can bypass Hadoop here, or at least have something like this
Twitter > Streaming Framework > HDFS
.. > Other DB
... > Spark
Regarding running locally or not, as long as you are fine with paying for idle hours on a cloud provider, go ahead.