I have the list of metrics those will be coming from graphite.
I want to seed checks for these graphite metrics in the Cabot database.
So when I open the graphite UI first time, I should see my seeded graphite checks.
For this which database tables I need to change?
Related
I am working on some project where i need to generate lead time for changes per application, per day..
Is there any prometheus metric that provides lead time for changes ? and How we integrate it into a grafana dashboard?
There is not going to be a metric or dashboard out of the box for this, the way I would approach this problem is:
You will need to instrument your deployment code with the prometheus client library of your choice. The deployment code will need to grab the commit time, assuming you are using git, you can use git log filtered to the folder that your application is in.
Now that you have the commit date, you can do a date diff between that and the current time (after the app has been deployed to PRD) to get the lead time of X seconds.
To get it into prometheus, use the node_exporter (or windows_exporter) and their textfile collectors to read textfiles that your deployment code writes and surface them for prometheus to scrape. Most of the client libraries have logic to help you write these files, and even if there is not, the format of the textfiles is pretty easy to use by writing the files directly.
You will want to surface this as a gauge metric, and have a label to indicate which application was deployed. The end result will be a single metric that you can query from grafana or set up alerts that will work for any application/folder that you deploy. To mimic the dashboard that you linked to, I am pretty sure you will want to use the over_time functions.
I also want to note that it might be easier for you to store the deployment/lead time in a sql database/something other than prometheus and use that as a data source into grafana. For applications that do not deploy frequently you would easily run into missing series when querying by using prometheus as a datastore, and the overhead of setting up the node_exporters and the logic to manage the textfiles might outweigh the benefits if you can just INSERT into a sql table.
I am supporting Filenet Applications and generally focus on performance improvement techniques. Often we face this issue related to the queries optimization. Generally we get the queries from DBA and these are DB SQL which are fired at the database level. Now from the application code we pass the CE SQL and not the DB SQL. I am aware that the CE parse the CE SQL to underlying DB SQL. I am trying to figure out if I have the DB SQL can I get the corresponding CE SQL which is being fired. A code or script which I can write in which I enter the CE SQL and the corresponding DB SQL gets generated. Appreciate if I could get any pointers on this as I am really stuck.
You need to enable Trace Logging for the DB subsystem. This is done through the Trace Control tab of Domain configuration in ACCE. Then you will be able to see database queries in p8_server_trace.log.
For convenience you might want to enable tracing for the SRCH subsystem as well. Then original and generated queries will go hand in hand.
Detailed info on Trace Logging is available in the FileNet P8 documentation.
The way to capture CE SQL queries is to turn on auditing for the object class your are interested in and select Query Event as the event. Now every time a query is performed an event object is created. This object has a property called QueryText which contains the CE query that is performed. You could use the creation time or some other information in the query to match it to your database query.
The query events can be queried using the ACCE or accessed programmatically using the API object com.filenet.api.events.QueryEvent.
Be aware that on a busy system a lot of query events can be generated!
I try to write batch data to influxdb but with deferent retention policies, but I could not find a way to do it without grouping the batch data with same retention then send each of the data in different batch..
It is currently not possible to write data with different retention policies in the same batch.
I have a service that analyzes e-mails in a list. Each lists can contain 500 to 200,000 emails. The best way I found was to analyze these lists is using Sidekiq services background. It's working, but i don't think this is the best way to analyze these e-mails, the reason is :
The Sidekiq is configured to use threads 25, and i'm using MongoDB to store these emails.
The workflow of the job is:
Search the e-mail in MongoDB using the ID passed via parameters to the job;
Analyze the e-mail and update the status of the analysis in the database : Valid or Invalid.
So it makes an average of 25 concurrent accesses to the database and after 1 ~ 4 seconds (processing time) it need to access the database again to update the status ( 1 time for each job ).
If a list has 100.000 emails, 100.000 jobs will be needed to complete this task. 200.000 accesses will be made in the database, 25 concurrency in most of time.
This is a good way to perform this service ? Is there a way to perform less access to MongoDB ?
I can not see another way to do this .
I want to store webpages fetched by a web crawler. I don't have any random access. so whenever i want to read the stored data, i read from the start to the end.
We have tried solutions like HBase but one of the most good things about HBase is random access to records which we don't need at all. HBase has not proved to be stable for us after 1.5 years of test.
I want just a stack or queue on top of HDFS becuase the number of webpages is about 1 billion. I don't even want the queue behaviour of ActiveMQ i just want to be able to store the webpages so that i can read them all in case of a failure.
I don't want to use Files because i don't want to handle things like file rotations, file consistencies and ...
It is worth to mention that we need HDFS so we can run MapReduce jobs on the data when we want to send all the stored data to a solr cluster and to have good things like redundancy and availability by HDFS.
Is there a service on HDFS that just stores JMS records without any functionality for random access and without transparent view of records?