Im new here, I have an issue with one plugin,
So im using telegraf to get data from ipmi sensor, and it’s working I can see that in Grafana dashboard via InfluxDB,
so until here everything is working correctly,
I added another input plugin to my telegraf.conf : telegraf-speedtest/speedtest.conf at master · Derek-K/telegraf-speedtest · GitHub
Once im checking telegraf -test , I can see that the first plugin(ipmi sensor) is OK and second plugin (speedtest) is OK as well.
But speedtest measurements are not stored in the influxdb I check it using
root#d5c51db15460:/# influx -execute ‘show measurements’ -database ‘telegraph’
name: measurements
name
ipmi_sensor
Here as you can see there is only the ipmi_sensor :( .
Telegraf is restarted already, and both plugins are working with -test
im not sure where is the issue, I appreciate your help guys
thank you
The telegraf --test option does not send data to outputs. I would suggest using the --once option if you want to ensure data is making it to outputs as well.
Related
I am running an Airflow instance using Docker. I am able to access the Airflow UI using http://localhost:8080/. Also able to execute a sample dag using PythonOperator. Using PythonOperator I am able to query a big query table on GCP environment. The service account key JSON file is added in my docker compose yaml file.
This works perfectly.
Now I want to use BigQueryOperator and BigQueryCheckOperator for which I need a connection ID. This connection ID would come from Airflow connections which happens through Airflow UI.
But when I am trying to create a new Google Bigquery connection getting errors. Could anyone please help me to fix this.
In your docker compose file, can you set the environment variable GOOGLE_APPLICATION_CREDENTIALS to /opt/airflow/configs/kairos-aggs-airflow-local-bq-connection.json? This might be enough to fix your first screenshot.
Looking at the docs and comparing your second screenshot, I think you could try selecting 'Google Cloud Platform' as the connection type and adding a project ID and Scopes to the form.
The answers to this question may also be helpful.
I have set up JMeter in Non-GUI mode in docker with influxdb and Grafana.
Jmeter real time results are first shipped in influxdb and shows on Grafana dashboard. Below is the table format which is being shipped in influxdb:
My team requirement is to see the view result tree result in Grafana as they want to see request/response for each transaction as per below:
Please help me to ship the result in influxdb of same as we team want to see each request/response in Grafana otherwise no use of dockerize the Jmeter. I am already using influxdbbackendlistener but unable to get tree result in influxdb.
I the last weeks I have completed the Principal Hyperledger-Composer tutorials and I have created a private network for testing some features.
I have noticed that if you close your PC or stop the fabric (./stopFabric), and restart the private-network, it will be empty. You lose every action you perform in the network using the rest-server (add client, transaction history, etc).
Is there a way for saving my actions? Do I have to use a "multiple-organization-network" (https://hyperledger.github.io/composer/tutorials/deploy-to-fabric-multi-org) for maintaining the status of my data?
I have to maintain always online a peer for not loosing my data?
please read How to restart the fabric composer without losing the existing data? - you need locate the correct docker-compose.yaml file(s) and (from the directory or providing the path the appropriate command below) preserve them using:
Run docker-compose stop to stop the Fabric Containers
Run docker-compose start to restart where you left off.
I am currently trying to break into Data engineering and I figured the best way to do this was to get a basic understanding of the Hadoop stack(played around with Cloudera quickstart VM/went through tutorial) and then try to build my own project. I want to build a data pipeline that ingests twitter data, store it in HDFS or HBASE, and then run some sort of analytics on the stored data. I would also prefer that I use real time streaming data, not historical/batch data. My data flow would look like this:
Twitter Stream API --> Flume --> HDFS --> Spark/MapReduce --> Some DB
Does this look like a good way to bring in my data and analyze it?
Also, how would you guys recommend I host/store all this?
Would it be better to have one instance on AWS ec2 for hadoop to run on? or should I run it all in a local vm on my desktop?
I plan to have only one node cluster to start.
First of all, Spark Streaming can read from Twitter, and in CDH, I believe that is the streaming framework of choice.
Your pipeline is reasonable, though I might suggest using Apache NiFi (which is in the Hortonworks HDF distribution), or Streamsets, which is installable in CDH easily, from what I understand.
Note, these are running completely independently of Hadoop. Hint: Docker works great with them. HDFS and YARN are really the only complex components that I would rely on a pre-configured VM for.
Both Nifi and Streamsets give you a drop and drop UI for hooking Twitter to HDFS and "other DB".
Flume can work, and one pipeline is easy, but it just hasn't matured at the level of the other streaming platforms. Personally, I like a Logstash -> Kafka -> Spark Streaming pipeline better, for example because Logstash configuration files are nicer to work with (Twitter plugin builtin). And Kafka works with a bunch of tools.
You could also try out Kafka with Kafka Connect, or use Apache Flink for the whole pipeline.
Primary takeaway, you can bypass Hadoop here, or at least have something like this
Twitter > Streaming Framework > HDFS
.. > Other DB
... > Spark
Regarding running locally or not, as long as you are fine with paying for idle hours on a cloud provider, go ahead.
I want to create a social network in django framework in Openshift then I need at least a graph db (like Neo4j)and a relational db (like Mysql). I had trouble in add Neo4j to my project because openshift has not any cartridge for it. then I decide to install it with DIY, but I don't understand the functionality of start and stop files in .openshift/action hooks.Then I doing the following steps to install neo4j on server:
1.ssh to my account:
ssh 1238716...#something-prolife.rhcloud.com
2.go in a folder that have permission to write (I go to app-root/repo/ and mkdir test in it) and download the neo4j package from here. and extract it to the test folder that I created before :
tar -xvzf neo4j-community-1.9.4-unix.tar.gz
3.and finally run the neo4j file and start it:
neo4j-community-1.9.4/bin/neo4j start
but I see these logs and can't run the neo4j:
process [3898]... waiting for server to be ready............ Failed
to start within 120 seconds.
Neo4j Server may have failed to start, please check the logs.
how can I run this database in openshift ? where I am wrong ? and where is the logs in please check the logs?
I've developed an openshift cartridge that fixes the permission issue in openshift. I had to change the class HostBoundSocketFactory and SimpleAppServer in neo4j just to bind without using the 0 port, but using an openshift available port.
You can check at: https://github.com/danielnatali/openshift-neo4j-cartridge
it's working for me.
I would also not place it in the app-root/repo but instead I would put it in app-root/data.
You also need to use the IP of the gear - I think the env. variable is something like OPENSHIFT_INTERAL_IP. 127.0.0.1 is not available for binding but I think the ports should be open.
There are 2 ways neo4j can run : embedded or standalone(exposed via a rest service).
Standalone is what you are trying to do. I think the right way to setup neo4j would be by writing a cartridge for openshift, and then add the cartridge to your gear. There has been some discussion about this, but it seems that nobody has taken the time to do this. Check https://www.openshift.com/forums/openshift/neo4j-cartridge. If you decide to write your own cartridge, i might assist. Here are the docs: https://www.openshift.com/developers/download-cartridges.
The other option is running in embedded mode, which i have used. You need to set up a Java EE application(because neo4j embedded mode libraries are available only with java), and put the neo4j libraries in your project. Then, you would expose some routes, check for parameters and run your neo4j queries inside the servlets.