We use Apache Nifi for our data ingestion.
We extract data from multiple sources, especially oracle databases. A lot of the extracted tables contain timestamps. As we are working in an international context, I want to change the timezone of all the timestamps into UTC time.
how can I change the timezone of a timestamp column into UTC time?
is there a generic solution, so that all timestamps are converted at once?
You can able to convert timestamp column into UTC time.
Please look at below reference website.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#format
Just consider you are having attribute named "time" which having value 1420058163264
${time:format("yyyy/MM/dd HH:mm:ss.SSS'Z'", "GMT")}
In format method you can pass your timezone which can convert timestamp into UTC Timezone.
I hope that it helps/
Thanks
Related
I have to store an "event time" that could be in any timezone. By default rails stores these in the DB as UTC and strips off our custom timezone. When we read the attribute back it is converted to our application's timezone - we lose the original timezone information.
We're working around this by storing the timestamp (as UTC) and an offset (an integer number of seconds) which is fine, but I'd really love to be able to store this as just one column and have rails use that rather than converting to UTC on write and to the application's timezone on read.
Is having rails store the timestamp in non-UTC just for this one field just not possible?
I have table which have a datetime field named date. When doing a POST in order to insert a new row, the date sent from the client (browser) looks like 2015-11-20T14:30:00+10:00 which is actually a correct date and timezone.
However, inside Postgres this date has been inserted as 2015-11-20 04:30:00.000000, which as you can see, is not at all the same as above. I know the problem is related to the timezone. But I cannot seems to figure out a fix.
For information, I have configured my app timezone :
class Application < Rails::Application
config.time_zone = 'Brisbane'
end
Ideas?
2015-11-20T14:30:00+10:00 means that the local time of 14:30 is 10 hours ahead of UTC. Your database field reflects the correct UTC value of 04:30. This is often the desired behavior, especially if the value represent a timestamp - the date and time something occured (past tense).
In PostgreSQL, there are two different types of timestamp fields (reference)
The TIMESTAMP WITH TIME ZONE field accepts an input that contains a time zone offset. It then converts the value to UTC for storage. On retrieval, it uses the session's timezone setting.
The TIMESTAMP, or TIMESTAMP WITHOUT TIME ZONE simply stores the date and time given, ignoring any offset, and not converting to UTC.
Most of the time, you should indeed use TIMESTAMP WITH TIME ZONE. You should only use TIMESTAMP WITHOUT TIME ZONE if you need to retain the local date and time value, such as in scheduling of future events and calculation of business hours. And for those scenarios, it often makes more sense to split date and time into separate DATE and TIME fields.
One last thing - if you can avoid it, avoid using Rails time zones and use standard tzdb zones. "Australia/Brisbane" is the full tzdb identifier equivalent to the Rails "Brisbane" time zone. Refer to the section on Rails time zones at the bottom of the timezone tag wiki.
I found this gem to be incredibly useful and easy for correctly setting the time https://github.com/kbaum/browser-timezone-rails
I having an column of UNIX time stamp in my database table, which comes from a system that is in the Kuwait time zone.
My database server's time zone is Eastern Time US & Canada. Now I need to convert the UNIX time stamp in to Kuwait time zone date value using an SQL query.
Can anyone tell me how I can convert this UNIX time stamp into a Kuwait time zone date value?
Unix timestamps are integer number of seconds since Jan 1st 1970 UTC.
Assuming you mean you have an integer column in your database with this number, then the time zone of your database server is irrelevant.
First convert the timestamp to a datetime type:
SELECT DATEADD(second, yourTimeStamp, '1970-01-01')
This will be the UTC datetime that corresponds to your timestamp.
Then you need to know how to adjust this value to your target time zone. In much of the world, a single zone can have multiple offsets, due to Daylight Saving Time.
Unfortunately, SQL Server has no ability to work work time zones directly. So if you were, for example, using US Pacific time, you would have no way of knowing if you should subtract 7 hours or 8 hours. Other databases (Oracle, Postgres, MySql, etc.) have built-in ways to handle this, but alas, SQL Server does not. So if you are looking for a general purpose solution, you will need to do one of the following:
Import time zone data into a table, and maintain that table as time zone rules change. Use that table with a bunch of custom logic to resolve the offset for a particular date.
Use xp_regread to get at the Windows registry keys that contain time zone data, and again use a bunch of custom logic to resolve the offset for a particular date. Of course, xp_regread is a bad thing to do, requires certain permissions granted, and is not supported or document.
Write a SQLCLR function that uses the TimeZoneInfo class in .Net. Unfortunately, this requires an "unsafe" SQLCLR assembly, and might cause bad things to happen.
IMHO, none of these approaches are very good, and there is no good solution to doing this directly in SQL. The best solution would be to return the UTC value (either the original integer, or the datetime at UTC) to your calling application code, and do the timezone conversion there instead (with, for example, TimeZoneInfo in .Net or similar mechanisms in other platforms).
HOWEVER - you have lucked out in that Kuwait is (and always has been) in a zone that does not change for Daylight Saving Time. It has always been UTC+03:00. So you can simply add three hours and return the result:
SELECT DATEADD(hour, 3, DATEADD(second, yourTimeStamp, '1970-01-01'))
But do recognize that this is not a general purpose solution that will work in any time zone.
If you wanted, you could return one of the other SQL data types, such as datetimeoffset, but this will only help you reflect that the value is three hours offset to whomever might look at it. It won't make the conversion process any different or better.
Updated Answer
I've created a project for supporting time zones in SQL Server. You can install it from here. Then you can simply convert like so:
SELECT Tzdb.UtcToLocal('2015-07-01 00:00:00', 'Asia/Kuwait')
You can use any time zone from the IANA tz database, including those that use daylight saving time.
You can still use the method I showed above to convert from a unix timestamp. Putting them both together:
SELECT Tzdb.UtcToLocal(DATEADD(second, yourTimeStamp, '1970-01-01'), 'Asia/Kuwait')
Updated Again
With SQL Server 2016, there is now built-in support for time zones with the AT TIME ZONE statement. This is also available in Azure SQL Database (v12).
SELECT DATEADD(second, yourTimeStamp, '1970-01-01') AT TIME ZONE 'Arab Standard Time'
More examples in this announcement.
Ive noticed that in a model the created_at attribute when rendering a model to json has the format like:
2013-05-20T22:12:31Z
and calling Time.now and rendering to json gives a format like:
2013-05-21 00:04:03 +0000
I am trying to create a web service for an iOS app where I need to compare the current time vs the created time but cant do so because of the timezone difference.
What does the Z in the Created_at attribute represent? Is that a timezone? If so how do I change the Time.now timezone to match?
If Z is not a timezone, any recommendations?
ActiveRecord Timestamp columns (like created_at and updated_at) are loaded as the type ActiveSupport::TimeWithZone, while Time.now produces a Time. These classes have different default to_s formats, which accounts for your discrepancy.
As you guessed, both the Z and the +0000 relate to the timezone of the datetime. Your created at value is being expressed as an ISO 8601 date where Z means Zulu, another way of expressing the UTC or GMT 0-offset timezone. Good news is that means both of your dates are already in UTC.
I'm not sure what your use case is, but you should consider doing this date comparison on the server if possible instead of in the iOS client. In that case the string representation wouldn't matter since the underlying numerical values of the dates would be compared.
If you really need to do the comparison on the iOS client, then you need to explicitly format both date values when rendering the JSON so you will have a consistent representation. You'll want to do a created.strftime(...) and Time.now.strftime(...) to format the values. ISO 8601 is a good choice for date formats in an API, although you will probably want to check and see what's easiest to consume given the iOS libraries you are using.
If one only works with dates and no time, would it still be wise to save all your dates in UTC for a multi timezone app?
I suggest using DateTimeOffset - this allows you to store both the DateTime and the UTC offset.
Also, read this SO question about timezone best practices.
The general the answer is yes, but it will introduce errors in time conversion if time part is not stored. Timezone conversions need time part to produce meaningful results. So just store full UTC time (i.e. date and time), since you can always trim it to just date if necessary.