Influx DB - First / Last values - influxdb
I'm selecting first and last values from last minute from an influx db timeseries, here are requests :
echo -e 'LAST VALUE' && \
curl -G 'http://db_address:8086/query?q=select+last(value)+from+ambient_temperature+where+time>now()+-+1m&db=home_data' && \
echo -e '\nFIRST VALUE' && \
curl -G 'http://db_address:8086/query?q=select+first(value)+from+ambient_temperature+where+time>now()+-+1m&db=home_data' && \
echo -e '\nlast values' && \
curl -G 'http://db_address:8086/query?q=select+*+from+ambient_temperature+where+time>now()+-+2m&db=home_data'
LAST VALUE
{"results":[{"series":[{"name":"ambient_temperature","columns":["time","last"],"values":[["2016-05-19T08:43:09.645833512Z",20.71875]]}]}]}
FIRST VALUE
{"results":[{"series":[{"name":"ambient_temperature","columns":["time","first"],"values":[["2016-05-19T08:43:09.700503355Z",20.71875]]}]}]}
last values
{"results":[{"series":[{"name":"ambient_temperature","columns":["time","value"],"values":[["2016-05-19T08:42:11.69Z",20.71875],["2016-05-19T08:42:13.715Z",20.71875],["2016-05-19T08:42:15.695 Z",20.71875],["2016-05-19T08:42:17.705Z",20.71875],["2016-05-19T08:42:19.714Z",20.71875],["2016-05-19T08:42:21.71Z",20.71875],["2016-05-19T08:42:23.719Z",20.71875],["2016-05-19T08:42:25.729Z ",20.71875],["2016-05-19T08:42:27.724Z",20.71875],["2016-05-19T08:42:29.734Z",20.71875],["2016-05-19T08:42:31.788Z",20.71875],["2016-05-19T08:42:33.783Z",20.71875],["2016-05-19T08:42:35.764Z ",20.71875],["2016-05-19T08:42:37.758Z",20.71875],["2016-05-19T08:42:39.753Z",20.71875],["2016-05-19T08:42:41.793Z",20.71875],["2016-05-19T08:42:43.772Z",20.71875],["2016-05-19T08:42:45.782Z ",20.71875],["2016-05-19T08:42:47.777Z",20.71875],["2016-05-19T08:42:49.787Z",20.71875],["2016-05-19T08:42:51.782Z",20.71875],["2016-05-19T08:42:53.792Z",20.71875],["2016-05-19T08:42:55.846Z ",20.71875],["2016-05-19T08:42:57.796Z",20.71875],["2016-05-19T08:42:59.807Z",20.71875],["2016-05-19T08:43:01.816Z",20.71875],["2016-05-19T08:43:03.841Z",20.71875],["2016-05-19T08:43:05.851Z ",20.71875],["2016-05-19T08:43:07.831Z",20.71875],["2016-05-19T08:43:09.855Z",20.71875],["2016-05-19T08:43:11.865Z",20.71875],["2016-05-19T08:43:13.86Z",20.71875],["2016-05-19T08:43:15.84Z", 20.71875],["2016-05-19T08:43:17.85Z",20.71875],["2016-05-19T08:43:19.859Z",20.71875],["2016-05-19T08:43:21.869Z",20.71875],["2016-05-19T08:43:23.864Z",20.71875],["2016-05-19T08:43:25.874Z",2 0.71875],["2016-05-19T08:43:27.869Z",20.71875],["2016-05-19T08:43:29.879Z",20.71875],["2016-05-19T08:43:31.919Z",20.71875],["2016-05-19T08:43:33.883Z",20.71875],["2016-05-19T08:43:35.893Z",2 0.71875],["2016-05-19T08:43:37.903Z",20.71875],["2016-05-19T08:43:39.899Z",20.71875],["2016-05-19T08:43:41.908Z",20.71875],["2016-05-19T08:43:43.918Z",20.71875],["2016-05-19T08:43:45.913Z",2 0.71875],["2016-05-19T08:43:47.937Z",20.71875],["2016-05-19T08:43:49.932Z",20.71875],["2016-05-19T08:43:51.942Z",20.71875],["2016-05-19T08:43:53.937Z",20.71875],["2016-05-19T08:43:55.947Z",2 0.71875],["2016-05-19T08:43:57.972Z",20.71875],["2016-05-19T08:43:59.981Z",20.71875],["2016-05-19T08:44:01.976Z",20.71875],["2016-05-19T08:44:03.971Z",20.71875],["2016-05-19T08:44:05.981Z",2 0.71875],["2016-05-19T08:44:07.991Z",20.71875]]}]}]}
But I do not understand returned results, for last and first values I get a time that does not match any time in my series (see last request that queries values from last 2 minutes), I should have a time:
2016-05-19T08:42:11.69Z for first element
2016-05-19T08:44:07.991Z for last element
Where do first and last values time come from?
Why first and last value are not correnct?
Here is response from influxdb :
https://github.com/influxdata/influxdb/issues/6684
Related
delete data from influxdb 2.0
I want to delete data from influxdb v2.0: I read its doc and I try 2 ways that it says, but I get error. https://docs.influxdata.com/influxdb/v2.0/write-data/delete-data/ in cli: influx delete \ --host HOST \ --org ORG \ --token TOKEN \ --bucket BUCKET \ --start 2021-06-01T00:00:00Z \ --stop 2021-06-01T01:00:00Z error: Error: Failed to delete data: Not implemented. See 'influx delete -h' for help Can you help me, how can I delete data?
Delete data from same host:
influx delete --bucket example-bucket \
--start 2020-03-01T00:00:00Z \
--stop 2020-11-14T00:00:00Z
You can also delete data via Curl
curl --request POST https://influxurl/api/v2/delete?org=example-org&bucket=example-bucket \
--header 'Authorization: Token YOUR_API_TOKEN' \
--header 'Content-Type: application/json' \
--data '{
"start": "2022-01-19T06:32:00Z",
"stop": "2022-01-19T06:33:00Z",
"predicate": "_measurement=\"example-measurement\" AND feature=\"temp2\" "
}'
Predicate method not working properly. (bug)
For local CLI in docker container. I had multiple measurements in one bucket, so to delete only them using predicate is necessary. I 've put one day before the first measurement in start and one day after last measurment in stop influx delete --bucket home \ --start 2021-04-01T00:00:00Z \ --stop 2023-01-12T00:00:00Z \ --predicate '_measurement="personal_rating6"'
Use xmlstarlet to convert XML containing repeated and missing fields to tab delimited
I have a large complex XML file containing a pattern like the following <?xml version="1.0" encoding="UTF-8"?> <records> <record> <field1>field1</field1> <field2>field2</field2> <field2>field2</field2> <field3>field3</field3> <field4>field4</field4> <field4>field4</field4> </record> <record> <field1>field1</field1> <field1>field1</field1> <field3>field3</field3> <field4>field4</field4> <field4>field4</field4> </record> </records> I would like to use xmlstarlet to convert it to tab delimited with repeated fields subdelimited with a semicolon, e.g. field1\tfield2;field2\tfield3\tfield4;field4 field1;field1\t\tfield3\t\field4;field4 I can do what I need by collapsing repeated fields with a string processing routine before feeding the file to xmlstarlet but that feels hacky. Is there an elegant way to do it all in xmlstarlet?
It's been a while since you asked. Nevertheless...
Using xmlstarlet version 1.6.1 to extract and sort field names
to determine field order, and then produce tab-separated values:
xmlstarlet sel \
-N set="http://exslt.org/sets" -N str="http://exslt.org/strings" \
-T -t \
--var fldsep="'$(printf "\t")'" \
--var subsep="';'" \
--var allfld \
-m '*/record/*' \
-v 'name()' --nl \
-b \
-b \
--var uniqfld='set:distinct(str:split(normalize-space($allfld)))' \
-m '*/record' \
--var rec='.' \
-m '$uniqfld' \
--sort 'A:T:-' '.' \
-v 'substring($fldsep,1,position()!=1)' \
-m '$rec/*[local-name() = current()]' \
-v 'substring($subsep,1,position()!=1)' \
-v '.' \
-b \
-b \
--nl < file.xml
EDIT: --sort moved from $allfld to -m $uniqfld.
where:
all field names in input are collected in the $allfld variable
the exslt functions set:distinct
and str:split
are used to create a nodeset of unique field names from $allfld
the $uniqfld nodeset determines field output order
repeated fields are output in document order here but -m '$rec/*[…]'
accepts a --sort clause
the substring expression emits a separator when position() != 1
Given the following input, which is different from OP's,
<?xml version="1.0" encoding="UTF-8"?>
<records>
<record>
<field2>r1-f2A</field2>
<field2>r1-f2B</field2>
<field3>r1-f3</field3>
<field4>r1-f4A</field4>
<field4>r1-f4B</field4>
<field6>r1-f6</field6>
</record>
<record>
<field1>r2-f1A</field1>
<field1>r2-f1B</field1>
<field3/>
<field4>r2-f4A</field4>
<field4>r2-f4B</field4>
<field5>r2-f5</field5>
<field5>r2-f5</field5>
</record>
<record>
<field6>r3-f6</field6>
<field4>r3-f4</field4>
<field2>r3-f2B</field2>
<field2>r3-f2A</field2>
</record>
</records>
output becomes:
r1-f2A;r1-f2B r1-f3 r1-f4A;r1-f4B r1-f6
r2-f1A;r2-f1B r2-f4A;r2-f4B r2-f5;r2-f5
r3-f2B;r3-f2A r3-f4 r3-f6
or, piped through sed -n l to show non-printables,
\tr1-f2A;r1-f2B\tr1-f3\tr1-f4A;r1-f4B\t\tr1-f6$
r2-f1A;r2-f1B\t\t\tr2-f4A;r2-f4B\tr2-f5;r2-f5\t$
\tr3-f2B;r3-f2A\t\tr3-f4\t\tr3-f6$
Using a custom field output order things can be reduced to:
xmlstarlet sel -T -t \
--var fldsep="'$(printf "\t")'" \
--var subsep="';'" \
-m '*/record' \
--var rec='.' \
-m 'str:split("field6 field4 field2 field1 field3 field5")' \
-v 'substring($fldsep,1,position()!=1)' \
-m '$rec/*[local-name() = current()]' \
-v 'substring($subsep,1,position()!=1)' \
-v '.' \
-b \
-b \
--nl < file.xml
again emitting repeated fields in document order in absence of --sort.
Note that using an EXSLT function in a --var clause requires the
corresponding namespace to be declared explicitly with -N to avoid
a runtime error (why?).
To list the generated XSLT 1.0 / EXSLT code add -C before the -t option.
To make one-liners of the formatted xmlstarlet commands above -
stripping line continuation chars and leading whitespace - pipe
them through this sed command:
sed -e ':1' -e 's/^[[:blank:]]*//;/\\$/!b;$b;N;s/\\\n[[:blank:]]*//;b1'
To list elements in the input file xmlstarlet's el command can be
useful, -u for unique:
xmlstarlet el -u file.xml
Output:
records
records/record
records/record/field1
records/record/field2
records/record/field3
records/record/field4
records/record/field5
records/record/field6
You can use the following xmlstarlet command:
xmlstarlet sel -t -m "/records/record" -m "*[starts-with(local-name(),field)]" -i "position()=1" -v "." --else -i "local-name() = local-name(preceding-sibling::*[1])" -v "concat(';',.)" --else -v "concat('\t',.)" -b -b -b -n input.xml
In pseudo-code, it represents something like this
For-each /records/record
For-each element-name that starts with field
If it's the first element, output the item
Else
If check if the current element name equals the preceding one
Then output ;Item
Else output \tItem
b does mean "break" out of loop or if clause
n outputs a newline
Its output is
field1\tfield2;field2\tfield3\tfield4;field4
field1;field1\tfield3\tfield4;field4
The above code differentiates on the base of element names. If you want to differentiate based on element values instead, use the following command:
xmlstarlet sel -t -m "/records/record" -m "*[starts-with(local-name(),field)]" -i "position()=1" -v "." --else -i ". = preceding-sibling::*[1]" -v "concat(';',.)" --else -v "concat('\t',.)" -b -b -b -n a.xml
With your example XML the output is the same (because field names and field values are identical).
Extract specific byte offset using tshark
I have a pcap of ICMP packets. I am trying to use tshark to extract the payload data so that I can extract a specific byte offset. The tshark documentation is highly convoluted, especially for me, a beginner. I've been searching around a lot and I'm trying to piece together a command for the purpose of my goal. I can run the following command: shark -r test.pcapng -Y icmp -z flow,icmp,network > output.bin But it only outputs the packet list as it were shown in Wireshark. For example, I am trying to extract the following byte offset from each packet (offset 22): How would I go about extracting a specific byte offset with tshark? EDIT: Issuing the following command only returns a portion of the payload data, how can I get all of it? tshark -r test.pcapng -Y "frame.number == 13" -T fields -e data -w output.bin
I've provided an answer over at https://ask.wireshark.org/question/14795/extract-specific-byte-offset-using-tshark/, but for convenience, I'll summarize the 2 possible solutions I provided here. In a nutshell: The highlighted byte in the image appears to be the TTL field of the IP header. If that's the field you're interested in, you can obtain it via: tshark -r test.pcapng -Y "frame.number == 13" -T fields -e ip.ttl -w output.bin If you're looking for a more general solution to print the 22nd byte of the packet, regardless of whether it's the ip.ttl field or not, then you can use a solution such as: tshark -r test.pcapng -Y "frame.number == 13" -x -w output.bin | grep "^0010" | cut -d ' ' -f 9 The 2nd solution above also illustrates how you can dump all the bytes; it's done using tshark's -x option.
Run cvb in mahout 0.8
The current Mahout 0.8-SNAPSHOT includes a Collapsed Variational Bayes (cvb) version for Topic Modeling and removed the Latent Dirichlet Analysis (lda) approach, because cvb can be parallelized way better. Unfortunately there is only documentation for lda on how to run an example and generate meaningful output. Thus, I want to: preprocess some texts correctly run the cvb0_local version of cvb inspect the results by looking at the top n words in each of the generated topics
So here are the subsequent Mahout commands I had to call in a linux shell to do it. $MAHOUT_HOME points to my mahout/bin folder. $MAHOUT_HOME/mahout seqdirectory \ -i path/to/directory/with/texts \ -o out/sequenced $MAHOUT_HOME/mahout seq2sparse -i out/sequenced \ -o out/sparseVectors \ --namedVector \ -wt tf $MAHOUT_HOME/mahout rowid \ -i out/sparseVectors/tf-vectors/ \ -o out/matrix $MAHOUT_HOME/mahout cvb0_local \ -i out/matrix/matrix \ -d out/sparseVectors/dictionary.file-0 \ -a 0.5 \ -top 4 -do out/cvb/do_out \ -to out/cvb/to_out Inspect the output by showing the top 10 words of each topic: $MAHOUT_HOME/mahout vectordump \ -i out/cvb/to_out \ --dictionary out/sparseVectors/dictionary.file-0 \ --dictionaryType sequencefile \ --vectorSize 10 \ -sort out/cvb/to_out
Thanks to JoKnopp for the detail commands. If you get: Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String you need to add the command line option "maxIterations": --maxIterations (-m) maxIterations I use -m 20 and it works refer to: https://issues.apache.org/jira/browse/MAHOUT-1141
How programmatically to get Jenkins user id?
I would like to pass the user id of the person who started a Jenkins job to a script. The output of 'env' indicates no environment variables are set and the output of 'ant -v' indicates no properties are set. How do I access the user id of the person that started a job? (I understand that triggers can start jobs, but for this job, it will always be a person).
To get the job executor:
curl -s "${BUILD_URL}/api/json" | \
python -c '\
import json; \
import sys; \
obj = json.loads(sys.stdin.read()); \
print [ \
cause["userId"] \
for action in obj["actions"] \
if "causes" in action \
for cause in action["causes"] \
if "userId" in cause][0];'
Also see How to set environment variables in Jenkins? which explains how to pass it into a script.
BUILD_START_USER=`curl ${JOB_URL}'lastBuild/api/json' | python -c 'import json,sys;obj=json.loads(sys.stdin.read());print obj["'"actions"'"][1]["'"causes"'"][0]["'"userId"'"]'`