I have a column from a CSV that stores datetime. I need to be able to do some simple calculations on the seconds of that datetime field. I want to know how I can convert the datetime field into the UNIX epoch timestamp using a derived column.
In SQL I would usually just do something like this:
DATEDIFF(second,{d '1970-01-01'},myDateTime)
Which gives me the integer value.
However I've tried the same in SSIS and get errors, why is this not correctly parsing?
Have you had a look at the SSIS documentation online for datediff? There tends to be usable examples in those docs.
Related
Date fields were indexed as string (import tool) and using apoc for parsing queries is good approach?
How it's affect performance?
What's the best approach?
Be aware that my initial load is from import tool
Thanks
in advance
I suggest you to store your dates as integers representing milliseconds since epoch (milliseconds passed since 01-01-1970 00:00:00:00). It has some advantages:
Easy to compare: You do not need to convert the dates before comparing since it is a simple number. So to verifiy if the birth date of a is greater to birth date of b you simply do a.birthDate > b.birthDate.
Without dependencies: You will not depend on any library to compare your dates.
Since you are storing your dates as an simple integer you can convert it in the front-end application and present it in any format you choose.
Also when you need you will use the APOC procedures to manipulate the timestamps stored in the database.
I am building a rails app, where the user picks up a date from a date picker and a time from the time picker. Both the date and time have been formatted using moment js to show the date and time in the following way:
moment().format('LL'); //January 23,2017
moment().format('LTS'); //1:17:54 PM
I read this answer with guidelines about selection of a proper column type.
Is there documentation for the Rails column types?
Ideally, I should be using :date, :time or :timestamp for this. But since the dates are formatted, should I be using :string instead?
Which would be the correct and appropriate column type to use in this situation?
If you want to store a time reference in your database you should use one of the types the database offers you. I'll explain this using MySQL (which is the one I have used the most) but the explanation should be similar in other database servers.
If you use a timestamp column you will be using just 4 bytes of storage, which is always a good new since it makes smaller indexes, uses less memory in temporal tables during the internal database operations and so on. However, timestamp has a smaller range than datetime so you will only be able to store values from year 1970 up to year 2038 more or less
If you use datetime you will be able to store a wider range (from year 1001 to year 9999) with the same precision (second). The bad consequence is that a higher range needs more memory, making it a bit slower.
There are some other differences between these two column types that don't fit in this answer, but you should keep an eye on before deciding.
If you use varchar, which is the default column type for text attributes in Ruby on Rails, you will be forced to convert from text to datetime and vice-versa every time you need to use that field. In addition, ordering or filtering on that column will be very inefficient because the database will need to convert all strings into dates before filtering or sorting, making it impossible to use indexes on that column.
If you need sub-second precision, you can use bigint to meet your requirements, as MySQL does not provide a date specific type for this purpose
In general, I recommend using timestamp if your application requirements fit the timestamp limitation. Otherwise, use datetime, but I strongly discourage you to use varchar for this purpose.
EDIT: Formatting
The way you store dates in database is completely different from the way you display it to the user. You can create a DateTime object using DateTime.new(year, month, day, hour, minute, second) and assign that object to your model. By the time you save it into database, ActiveRecord will be in charge of converting the DateTime object into the appropiate database format.
In order to display a value that is already stored in database in a specific format (in a view, API response, etc.) you can hava a look at other posts like this one.
You can have a timestamp column in your database, and then parse the request to a ruby datetime object like this:
d = Time.parse(params[:date])
t = Time.new(params[:time])
dt = DateTime.new(d.year, d.month, d.day, t.hour, t.min, t.sec, t.zone)
#now simply use dt to your datetime column
On Postgres you can save a ruby DateTime object straight into a postgres timestamp field, e.g
User.first.update_attribute('updated_at', dt )
Another option is to concatenate your date and time strings into one and then u can do a one-liner:
User.last.update_attribute('created_at', Time.parse('January 23,2017 1:17:54 PM'))
I'm pretty sure this will work on MySQL datetime or timestamp as well.
Credit to david grayson Ruby: combine Date and Time objects into a DateTime
I have a table with a column timestamp in type TIMESTAMP in BigQuery. When I display it on my console, I can see timestamps as follows: 2015-10-19 21:25:35 UTC
I then query my table using the BigQuery API, and when I display the result of the query, I notice that this timestamp has been converted in some kind of very big integer like 1.445289935E9 in string format.
Any idea on how do I convert it back to normal time? something I can use in my ruby code?
Time.at("1.468768144014E9".to_f)
I need to develop an InfluxDB Time Series. The time series needs to contain the following information:
time - The time the data was recorded. This will use the InfluxDB Time Field.
value - The value for the time series. A simple integer.
date - A separate date associated with the value. This date has nothing to do with the "time" field. It will be used to help narrow down queries.
My current line of thinking is to save the above "date" field as a separate "column" in the time series so that I can use the "where" clause to filter the data using that date. However I am struggling with how to represent that. Does InfluxDB support any kind of date or date/time fields? For the "time" field it seems to just use milliseconds. However if I try the same in a field with a different name, then the normal time queries don't work. So for example:
select * from myseries where time > now() - 1d
The above query will work just fine.
vs
select * from myseries where date > now() - 1d
This query will fail with an error because it doesn't seem to know how to treat "date" as a time value.
Is there a better representation for dates in this scenario?
InfluxDB data types can be only be one of: floats, ints, bools, or strings. The time field is a special exception.
You can use integers representing count-since-epoch for the date field. Nice convenient functions like now() don't seem work for that though (using v0.13):
insert test_dates date_int=1573405622000000000i,desc="years from now"
insert test_dates date_int=1373405661000000000i,desc="years ago"
Testing that with now()
select * from test_dates where date_int > now()
gives:
name: test_dates
time date_int desc
1473404302801938064 1573405622000000000 years from now
1473404315927493772 1373405661000000000 years ago
And:
select * from test_dates where date_int < now()
gives:
name: test_dates
time date_int desc
1473462286404084162 1573405622000000000 years from now
1473462286408231540 1373405661000000000 years ago
Seems every date_int is somehow both greater than and less than now()
So the comparison isn't a syntax error if you use integers, but doesn't work the way we'd like.
One way to solve this is to create your own date-to-int conversion in the front-end app. Then a date comparison on the front-end is an int comparison within InfluxDB. Clunky, but those are the data types we have.
The date as stored in InfluxDB could be a single epoch-based int, or store separate int fields for year, month, day within InfluxDB. The queries are bigger and slower in the latter case, but it's easier to read and debug.
Good luck!
Change this date format which is in sqlite db 12/10/11 to 12-10-11 (mm-dd-yy) I am unable to do so .I am a noob in sqlite and have to parse this value SELECT strftime('%d-%m-%Y',Date) from report but I am getting null as sqlite db excepts value in mm-dd-yy so How do I convert format 12/10/11 to 12-10-11 (mm-dd-yy) .Thanks in advance .Really appreciate the help.
The short answer:
If you have a text string stored as "12/10/11" that you want reported as "12-10-11", you should use the replace(X,Y,Z) function, to replace occurrences of Y in X with Z. Thus:
SELECT replace('12/24/11','/','-');
will return:
12-10-11
The long answer:
First, dates do not actually exist as a proper datatype in SQLite. They're stored as either TEXT, REAL, or INTEGER values. See date and time datatype in SQLite. So it depends upon how your date was stored in the database.
Second, you seem to be implying that you stored the date in a "mm/dd/yy" format. That's not a valid/useful TEXT format to be storing date/time values (as the date cannot be sorted, cannot used in "greater than" and "less than" operations, cannot be used in SQLite date functions, etc.). You really want to store datetime values in one of the formats listed in the "Time strings" section of the date and time functions document.
So, generally you should store your date/time values in one of those formats, use NSDateFormatter to convert that to a NSDate when you retrieve it from the database. And when you want to display the date value in your app, use whatever format you want for output.
But, if you don't care that the dates are stored as text strings and are not effectively usable as dates in SQLite, then just treat it as a plain old TEXT string and use TEXT functions, such as replace(X,Y,Z) to replace occurrences of "/" with "-", as outlined above.