If timestamp attribute is not assigned at publish time or read time, what timestamp is used for windowing? Documentation states that first time system sees each record. Is this system dataflow topology or Pub/Sub?
If timestampAttribute is not provided, the system will generate record
timestamps the first time it sees each record. All windowing will be
done relative to these timestamps.
The timestamp assigned to messages when you don't specify a timestamp attribute is the "publishTime" set on the pubsub message.
See: https://github.com/apache/beam/blob/50d0760faf01bdcdea988157a6b732bb448ba4b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java#L178
Related
I have two tables both of which have a timestamp field [TIMESTAMP_TZ] and when I perform a join based on this timestamp field the plan in snowflake DB shows an auto conversion on these timestamps into LTZ. Ex
(TO_TIMESTAMP_LTZ(CAG.LOAD_DATE_UTC) = TO_TIMESTAMP_LTZ(PIT.CSAT_AGREEMENT_LDTS))
Any reason why this is happening?
TIMESTAMP_TZ means your timestamp is linked to a time zone and TIMESTAMP_LTZ is your local timezone. Probably the timezones of your two timestamps are different and thus Snowflake converts them automatically to your local timezone to match them correctly.
Heading ##CALL ga.timetree.single({time: 1463659567468, create: true})
https://github.com/graphaware/neo4j-timetree
https://graphaware.com/neo4j/2014/08/20/graphaware-neo4j-timetree.html
The above link says that time is in long format YYYYMMDDHHmmss. But the time parameter doesn't make any sense and random nodes are getting generated in neo4j. enter image description here
What does the time parameter hold and what is the meaning of it?
The time parameter is a millisecond timestamp, or milliseconds elapsed since the UNIX epoch, which is an extremely common means of storing time-related data, you can find this in use in nearly every digital system.
The timestamp cited here represents "2016-05-19 12:06:07". The timetree built starts from a root (this is a modeling convenience), and then its child is the year (2016) followed by the month (5), then the date of the month (19). Looks like it didn't automatically create any nodes for time resolutions beyond that.
Keep in mind that now that Neo4j has native temporal values that you can use in Cypher and store as properties (as well as index), time trees are going to be less useful, as you can always do index lookups on indexed temporal properties.
There are still some cases where time trees can still be very useful, however, such as when you're searching for events that happened within some unit of time that disregards its parent units...such as finding events that happened on Mondays regardless of month, or on Januaries regardless of year, and so forth.
https://github.com/influxdata/telegraf/pull/1557
Apparently some people have been asking for this, and this Github PR is the closest thing I can find to a solution, but it was ultimately denied(I think?).
Basically, I have a JSON object I'm getting from Stackdriver, which includes a Timestamp in ISO8601, which I convert to Unix time. I can insert the entire JSON response into Influx fine, but the timestamp from Stackdriver appears as a tag for a series, rather than the index of the time series itself. As a result, it is unfeasible to query by Stackdriver's provided timestamp. I could simply just drop it, and use the Influx provided timestamp, but it is essentially querying incorrect/imprecise data.
Does anyone have a clever way to approach this?
tl;dr How can I use Telegraf to override InfluxDB's timestamps with my own timestamps?
I have a date based salt for our user password storage using Spring Security. The date is stored in MySQL without any timezone specific information. The salt function actually ends up using a java.sql.Timestamp during the salting process.
We are considering moving the server to another timezone and I'm wondering if this might affect the security check in some way
Update.
Ok, here's what I found
Hibernate returns a Timestamp in stead of a Date
Spring Security uses the toString method of the provided salt object
The toString method of Date includes a timezone, which can spell problem if you move your server and your salt is based on it.
However, the toString method of Timestamp doesn't include timezone information.
So, if you create your salt date object as a Timestamp, you're golden
However, the Timestamp returned by MySQL is truncated at second level, so I ended creating a Timestamp, truncating the nanoseconds and assigning it to my date.
I've updated my question with the answer
I have some daily analytics-style records that track usage on my site, and they work as follows:
When an 'event' occurs, the site looks for a record that was created at Time.now.midnight.
If such a record is not found, a new one is created, and created_at is set to Time.now.midnight.
Here's the question - does Time.now.midnight get recorded differently depending upon the client's time zone? If so, am I correct in assuming that the above very simple system will break down?
How can I fix it so all analytics records, irrespective of the time zone of the user who triggered their creation, get pegged to one time?
Note: I imagine that setting the created_at field to a Date instead of Datetime might have been better in this scenario. Let's assume we're stuck with datetime for this question.
Time.now does not get recorded differently based on the clients timezone.
Time.now returns a new time using the system timezone (aka the server)
To use a client specific timezone you have to have a user select their zone and use Time.current or Time.zone.now (which do the same thing)
created_at is usually pinned to UTC, so you shouldn't have any issues their either.
(to change this you need to change Rails.root/config/application.rb)
config.time_zone = "whatever you want the ActiveRecord default to be"