I'm am building a pipeline in Data Fusion where we use the Database Plugin to ingest data from our on-prem Oracle DB and insert into a BigQuery table. The Database Plugin correctly infers timestamp data types for date fields in our Oracle tables. The issue is, however, that the date fields are actually in MST timezone. Data Fusion, however, assumes they are in UTC.
Ex: Date in on-prem DB is Mar 11, 2020, 5:45:40 AM MST and it comes up as Mar 11, 2020, 5:45:40 AM UTC in BigQuery.
In the pipeline, I am using the Wrangler Plugin to transform column data types using directives. I tried using the parse-as-date DATE_COLUMN US/Mountain directive, but it did not work.
I have asked GCP support if there's a way to set default Data Fusion timezone to MST. I'm asking here to see if there's a way to do it with Plugins.
I don't think there's a way to set MST as the default time zone in Data Fusion; however, I tried to replicate the scenario and I was able to use parse-as-date DATE_COLUMN MST to parse the column and insert it into BigQuery with the correct time in UTC Mar 11, 2020, 11:45:40 AM UTC.
Related
We use Apache Nifi for our data ingestion.
We extract data from multiple sources, especially oracle databases. A lot of the extracted tables contain timestamps. As we are working in an international context, I want to change the timezone of all the timestamps into UTC time.
how can I change the timezone of a timestamp column into UTC time?
is there a generic solution, so that all timestamps are converted at once?
You can able to convert timestamp column into UTC time.
Please look at below reference website.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#format
Just consider you are having attribute named "time" which having value 1420058163264
${time:format("yyyy/MM/dd HH:mm:ss.SSS'Z'", "GMT")}
In format method you can pass your timezone which can convert timestamp into UTC Timezone.
I hope that it helps/
Thanks
I am using the VSTS REST API to get some information about the iterations of a specific project, specifically the method at this link: https://www.visualstudio.com/en-us/docs/integrate/extensions/reference/client/api/tfs/work/restclient/workhttpclient2_2#method_getTeamIterations
All of the information I'm requesting is there, but the start and finish dates for each iteration are being adjusted based on my local timezone. So an iteration that has an end date of today, Wed May 31 2017, will come back with a finishDate that looks like this:
Tue May 30 2017 19:00:00 GMT-0500 (Central Daylight Time)
I have some code that looks through these iterations using the current date to find which iteration is underway, so when it's run on the last day of an iteration, it's being tricked by this returned value into thinking that the next iteration has already started, or that we're outside of the actual current iteration.
Is this unintended behavior, or is there something I'm missing?
From my experience, this is expected behavior. TFS will convert your input date time values into UTC and store those value in database.
Likewise, when you are querying those date time values, it will convert back to your current timezone which is defined by the user context that you are using. Let say if you are using your credential then it's in your personal account settings (Click your profile in the top right corner, and there is a timezone setting).
In my rails application,users can create questions and publish that, anybody from any country can response for that.
We are designing database structure for that. so planning to get user timezone using some js and while answering converting that time and to store in a separate column(tz_created_at).
so in created at the date will be stored in utc format, and in another column say tz_created_at the datetime will be stored as user's timezone converted time. (ie) in created_at column i have
irb(main):013:0> DateTime.now.utc => Sun, 30 Mar 2014 18:54:46 +0000
And in tz_created_at column i have
irb(main):015:0> DateTime.now
+ 6.hours(from user's timezone) => Mon, 31 Mar 2014 06:58:31 +0600
we are using sunspot solr to show some statistics in response over the time period.
so while querying for responses for a particular question i will search in tz_created_at column. is this approach is correct . please correct me if i am wrong
Instead of doing that, you probably want to store the datetime as UTC, then store the user's timezone in another column. By doing that you can always change the timezone if you need to without affecting the true time (e.g. the user moves country). When you display a time to the user you can easily convert the time to their zone to display it.
If you're a subscriber of railscasts there's a video on it there, but if you're not a subscriber there's an older one that's free to watch that will give you a general idea of best practices.
It appears that when I use Firebase.ServerValue.TIMESTAMP currently, it is using PDT (I assume that will be changing to PST in the fall).
Is there a way to get this in UTC instead of PDT?
Firebase timestamps are always stored as milliseconds since the epoch (midnight of 1/1/1970 in UTC). This is the same way that dates work in Javascript and many other languages. This is a timezone-agnostic way of representing time.
Generally speaking, timezone only plays a role in how a time is displayed to a user, not in how it's represented under-the-hood. Firebase timestamps are no different.
So, if you construct a JS date object using a timestamp created by Firebase.ServerValue.TIMESTAMP, it will automatically have the same timezone as the machine on which it is being displayed.
I having an column of UNIX time stamp in my database table, which comes from a system that is in the Kuwait time zone.
My database server's time zone is Eastern Time US & Canada. Now I need to convert the UNIX time stamp in to Kuwait time zone date value using an SQL query.
Can anyone tell me how I can convert this UNIX time stamp into a Kuwait time zone date value?
Unix timestamps are integer number of seconds since Jan 1st 1970 UTC.
Assuming you mean you have an integer column in your database with this number, then the time zone of your database server is irrelevant.
First convert the timestamp to a datetime type:
SELECT DATEADD(second, yourTimeStamp, '1970-01-01')
This will be the UTC datetime that corresponds to your timestamp.
Then you need to know how to adjust this value to your target time zone. In much of the world, a single zone can have multiple offsets, due to Daylight Saving Time.
Unfortunately, SQL Server has no ability to work work time zones directly. So if you were, for example, using US Pacific time, you would have no way of knowing if you should subtract 7 hours or 8 hours. Other databases (Oracle, Postgres, MySql, etc.) have built-in ways to handle this, but alas, SQL Server does not. So if you are looking for a general purpose solution, you will need to do one of the following:
Import time zone data into a table, and maintain that table as time zone rules change. Use that table with a bunch of custom logic to resolve the offset for a particular date.
Use xp_regread to get at the Windows registry keys that contain time zone data, and again use a bunch of custom logic to resolve the offset for a particular date. Of course, xp_regread is a bad thing to do, requires certain permissions granted, and is not supported or document.
Write a SQLCLR function that uses the TimeZoneInfo class in .Net. Unfortunately, this requires an "unsafe" SQLCLR assembly, and might cause bad things to happen.
IMHO, none of these approaches are very good, and there is no good solution to doing this directly in SQL. The best solution would be to return the UTC value (either the original integer, or the datetime at UTC) to your calling application code, and do the timezone conversion there instead (with, for example, TimeZoneInfo in .Net or similar mechanisms in other platforms).
HOWEVER - you have lucked out in that Kuwait is (and always has been) in a zone that does not change for Daylight Saving Time. It has always been UTC+03:00. So you can simply add three hours and return the result:
SELECT DATEADD(hour, 3, DATEADD(second, yourTimeStamp, '1970-01-01'))
But do recognize that this is not a general purpose solution that will work in any time zone.
If you wanted, you could return one of the other SQL data types, such as datetimeoffset, but this will only help you reflect that the value is three hours offset to whomever might look at it. It won't make the conversion process any different or better.
Updated Answer
I've created a project for supporting time zones in SQL Server. You can install it from here. Then you can simply convert like so:
SELECT Tzdb.UtcToLocal('2015-07-01 00:00:00', 'Asia/Kuwait')
You can use any time zone from the IANA tz database, including those that use daylight saving time.
You can still use the method I showed above to convert from a unix timestamp. Putting them both together:
SELECT Tzdb.UtcToLocal(DATEADD(second, yourTimeStamp, '1970-01-01'), 'Asia/Kuwait')
Updated Again
With SQL Server 2016, there is now built-in support for time zones with the AT TIME ZONE statement. This is also available in Azure SQL Database (v12).
SELECT DATEADD(second, yourTimeStamp, '1970-01-01') AT TIME ZONE 'Arab Standard Time'
More examples in this announcement.