I use telegraf to send some data from a database to InfluxDB in regular intervals which works fine apart from one issue:
I need to replace telegraf's auto-generated timestamp (which is the current time at the moment of telegraf reading the data to transmit) with a field from the data.
(To answer the "why?" question: So the data I get in InfluxDB as a result actually matches the time of the event I want to record).
I would have thought there's some standard configuration parameter or an easy to find processor plugin, that let's me replace the default timestamp with the content of a field, but I didn't find any.
It does not seem to me to be a very exotic request and Telegraf's "Metric" does have a "SetTime" function, so I hope someone already solved that and can answer that.
You can use a starlark processor to accomplish this, at least if you are able to put the correct timestamp into a tag first.
Here an example of how I use a starlark processor to replace the timestamp of the measurement with the content of a tag that I already populated with the correct timestamp in the input plugin:
[[processors.starlark]]
source = '''
def apply(metric):
if "nanoseconds" in metric.tags:
metric.time = int(metric.tags["nanoseconds"])
return metric
'''
Due to the fact that
no one had an idea so far,
that feature isn't available in all plugins coming with telegraf, and
I have to solve this pretty urgently,
I wrote a telegraf processor plugin that does exactly what I needed.
(I will likely offer to contribute this to telegraf in the future, when I have a bit more time to breathe than right now.)
There exists a method to do this both for JSON format input and CSV input.
Here's a link to the JSON format description and how to set timestamps based on a value that's in the payload (to the format you can specify).
use
[[processors.date]]
tag_key = "name_of_column of your timestamp"
date_format = {time format applicable in Go language}
Related
I've got a lot of "duration" values - basically a race duration - in a format m:ss.millis [4:03.810 for example].
Currently GS handles it as text, but I would like to use those values for comparison and create some statistics.
Is this possible? I have read this: How to format a duration as HH:mm in the new Google sheets but even though I have created a custom formats like:
or
but neither with one nor with another I cannot use those values for calculations. GS always complains about the values beeing stored as text.
I guess I'm just doing something wrong, but I definitely need to be able to provide values in this format and be able to use them in calculations.
How can I do that?
I regret that Duration seems to be a useless abomination and Sheets appears to offer no relatively easy way to convert text representation to values. Pending a better answer I suggest you convert all your durations as below:
=(left(A1,find(":",A1)-1)+right(A1,6)/60)/1440
format as Number:
mm:ss.000
and then apply your formulae.
(Change , to ; if required by your locale.)
A shorter formula might be used to cajole TIMEVALUE to work by including an hour value of 0:
=TIMEVALUE("00:"&A1)
A happy coincidence brought me back here because only very recently have I found that Google Sheets does offer a way to convert Text to Number (or I was having another aberration when I claimed otherwise). However, this still seems not to apply to Duration. Maybe there is yet hope though.
I have millions of entries that are already in epoch ms format as longs.
I don't want to waste time converting to a string like "2016-..." if possible.
Does influx-python's InfluxDBClient have a precision arg, similar to how the native influx client allows -precision when it imports? I looked i github but didn't find it.
Or is the native client just converting everything into rfc3339 under the covers anyhow? I would think that the underlying wire protocol would want stamps as simple longs not strings.
I faced similar trouble. Its in documentation while writing points you can pass time_precision as parameter in the write_point function. If you are using series Helper look at
https://github.com/influxdata/influxdb-python/issues/502
Pull request I made. It helps in setting it in one place and not having to pass it everywhere. I opened up an issue for this.
https://github.com/influxdata/influxdb-python/issues/501
I can't find it specified anywhere. I did find a Microsoft example that had "5/5/1955". Is that d/m/y or y/m/d.
I'm guessing that I probably ought to use ISO 8601, but it would be nice to know for sure.
According to the OASIS spec, the type associated with URI
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/dateofbirth is xs:date, so it should be a normal XML date format... which is indeed ISO-8601, according to XML Schema Part 2. (That talks about the time zone for a date being representable, which strikes me as a little odd, but never mind.)
I'm trying to blast an 8-mer (string of length 8) against the NCBI database. However, whenever I use qblast, it comes up empty with respect to matches. This is my code:
from Bio.Blast.NCBIWWW import qblast
import Bio.Blast.NCBIXML as parser
a = qblast('blastp','nr','GGMPSGCS')
b = parser.read(a)
print b.alignments`
Whenever I do this, it just prints the empty list []. Why is this happening? Can anyone shine a light on it?
I can get a match using the NCBI online BLAST tool, and I can even get a match if I use a longer kmer like "SSRVQDGMGLYTARRVR". It just happens that all the 8-mers I search come up empty.
From the FAQ at http://biopython.org/DIST/docs/tutorial/Tutorial.html
Why doesn’t Bio.Blast.NCBIWWW.qblast() give the same results as the NCBI BLAST website?
You need to specify the same options – the NCBI often adjust the default settings on the >website, and they do not match the QBLAST defaults anymore. Check things like the gap >penalties and expectation threshold.
Check that qblast is using the same defaults, if not sure, make them explicit. I wouldn't be surprised if it's doing some sort of "read to short" filtering step.
As in this answer, you have to fine-tuning the qblast to override the defaults. The WWW frontend of NCBI-BLAST automatically adjusts your parameters to match the short (8 bp) sequences, but if you do it through Biopython API you have to do it manually.
I format dates with the command line option -f %Y-%m-%d or even %d-%b-%y
but each date comes out four years and one day ahead of the date I input
for example, date 01.06.2012 after parsing without -f option comes as 2016-06-02
toying with -f gives same result
What is the reason? Are there any workarounds,
except hardcode and substract back these 4 years and 1 day?
I am using xls2csv (by V.B.Wagner, comes with catdoc package in debian)
and switching to another parser can be very expensive option
Tools xls2csv is a Perl application that uses Spreadsheet::ParseExcel library.
Based on such library documentation, one of known problems is:
If Excel has date fields where the specified format is equal to the system-default for the short-date locale, Excel does not store the format, but defaults to an internal format which is system dependent. In these cases ParseExcel uses the date format 'yyyy-mm-dd'.
So you probably manipulate with Excel file that does not contain date formating due to above listed issue.
That's a known bug. A patch is available at
https://www.wagner.pp.ru/cgi-bin/cvstrac/catdoc/tktview?tn=14,4
It works.
By the way, there are two programs called xls2csv, we're talking about the one from the catdoc package, not the Perl program.