I am getting data from rest API in JSON forma and have a scenario where a column can have multiple date format. The current date format could be either 2011-02-12T01:00:00 or 2020-04-15T20:44:57.38or could be null or something else also.
I want to parse it through expression and trying to capture the full date string. The following expression seems to be working fine however it is truncating the millisecond part and returning value upto second only.
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'))
But when I tried it with millisecond usinf below expression:
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'))
It is throwing error:
TT_11132 Transformation [Expression3] had an error evaluating output column [JobStartDate_out].
Error message is [<<Expression Error>> [TO_DATE]: invalid string for converting to Date
... t:TO_DATE(u:TO_CHAR(t:<02/12/2011 01:00:00>,u:'MM/DD/YYYY HH24:MI:SS'),u:'MM/DD/YYYY HH24:MI:SS.MS')].
I searched few option using below but getting parsing error.
DECODE (TRUE,
iff(isnull(%date_milli%),
'\N',
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'),
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'),
ERROR('NOT A VALID DATE')))
What could be the possible resolution to handle the multiple date format in Informatica? Here JSON date format is string and I am mapping it to date/time type and using Output Marco Fields to combine multiple similar column together.
Why dont you check both options - with and without milliseconds?
You can use below iif condition. Also i think your expression has some issues.
I assumed date_milli is a character type. If its a date, then you can change below expressions accordingly.
iff(isnull(%date_milli%),null,
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS')
))
)
Related
I'm using CsvProvider to parse a CSV file i receive from an external source. The date format of the file is dd/MM/yy. But CsvProvider infers it as MM/dd/yy and get all dates wrong.
Is there any way I can pass the dateformat to CsvProvider?
I can read the dates as string and then convert them using Seq.Map to Datetime (which is two steps).
I'm trying to find out if i can do in a single step
Example
csv file sample.csv is having data like below
11/01/90
12/01/90
13/01/90
14/01/90
15/01/90
I'm using the below F# code
type MyCsvProvider = CsvProvider<Sample= "sample.csv", HasHeaders=false, Schema="Date (date)">
MyCsvProvider.Load("sample.csv")
The CsvProvider type provider does not have a way of explicitly specifying a format for parsing dates, so if you have some completely non-standard format, you'll just have to read it as string (which is what CSV provider infers if it cannot parse dates automatically) and then parse the date values explicitly.
That said, you can specify the Culture parameter, which makes it possible to parse dates in format that is common in countries outside of the US. For example, your date format would work fine with the en-GB culture (in the UK, you first write day, then month and then year).
In the following small example, the Test property is inferred as DateTime:
type A = CsvProvider<"""Test
11/01/90
12/01/90
13/01/90
14/01/90
15/01/90""",Culture="en-GB">
let r = A.GetSample().Rows |> Seq.head
r.Test
So Im pretty new to this stuff but working through a few issues. What I am trying to do is pull source files from a Flat File Source but the dates in all my source files are formatted to YYYYMMDD so I have inserted a Derived Column task and created an expression to format all the columns with dates YYYYMMDD to YYYY-MM-DD and that looks like this:
LEFT(ISSUE_DT,4) + "-" + SUBSTRING(ISSUE_DT,5,2) + "-" + RIGHT(ISSUE_DT,2)
All is good with that except it's in the data type of DT_WSTR so I dropped in a Columns Conversion task to convert DT_WSTR to DT_DATE but I keep getting the following error:
[Columns Conversion [1]] Error: Data conversion failed while converting
column "ISSUE_DT Formatted" (258) to column "Copy of ISSUE_DT Formatted"
(16). The conversion returned status value 2 and status text "The value
could not be converted because of a potential loss of data.".
I have tried opening the advanced editor and navigated to the Data Conversion Output Columns and tried changing the DataType under Data Type Properties to DT_DATE but still the same error..
What am I missing or doing wrong??
Data Flow
Formatted Dates
Column Conversion
Column Conversion Advanced Editor
You have some dates that are not in the format that your SSIS package is expecting. Perhaps single-digit months or days do not have a leading 0. That scenario would definitely cause this specific error.
Take today for example, if single-digit months or days did not have leading zeroes, you have 2018918 instead of 20180918. Without seeing the data, I can't guarantee that this is the exact issue, but it is something like this. The error is occurring when converting the string to the date. So continuing with the example above, after your Derived Column ISSUED_DT formatted would have a value of '2018-91-18' which of course is not a valid date and causes the error.
I'm attempting to periodically push data to a fusion table, and I have everything working except for the ability to insert the time portion of a DATETIME.
It works as expected if I omit the datetime field all together, e.g.:
service.query().sql(sql='INSERT INTO <my table> (Humidity, Temperature, Luminosity) VALUES (40, 70, 100)')
However, I get a synatx error every time I attempt to write the date. If I separate the date with /'s, it errors on those, so I switched to -'s, which seems to appease it. Only now, it chokes on the :'s that separate hour from minute:
service.query().sql(sql='INSERT INTO <my table> (Date, Humidity, Temperature, Luminosity) VALUES (04-12-2014 13:51, 40, 70, 100)')
yields:
HttpError: <HttpError 400 when requesting https://www.googleapis.com/fusiontables/v1/query?alt=json&sql=INSERT+INTO+(my table id)+%28Date%2C+Humidity%2C+Temperature%2C+Luminosity%29+VALUES+%2804-12-2014+13%3A51%2C+40%2C+70%2C+100%29 returned "Invalid query: Parse error near ':' (line 1, position 117).">
There's only one ':', so it's gotta be the time separator. Is there some other character I should be using to separate hours from minutes? Does it need to be escaped somehow? All of examples I can find in Google's documentation don't interact with DATETIME's.
When I tried without a ':' at all (e.g., 04-12-2014 1351), it didn't throw an error, but the date was interpreted in a wildly inaccurate way (09/01/1351).
Thanks!
Have you tried adding quotation marks around datetime?
Trying to retrieve a dateTime Information from Timesten Database and use it in Saxon Xquery .below is the example for that and getting the below error . Do we need to convert timesten dateTime to saxom dataTime if yes how to do that ? pls help me if have idea.
let $DateVar:=fn:data($PERSON/BIRTHDAY)
where as $PERSON/IN_BIRTHDAY is 2010-04-04 03:16:04.000000
if I am trying
let $day-b-DT :=day-from-dateTime($DateVar)
I am getting
Validation error
FORG0001: Invalid dateTime value "2010-04-04 03:16:04.000000" (Day must be two digits)
net.sf.saxon.s9api.SaxonApiUncheckedException: Invalid dateTime value "2010-04-04 03:16:04.000000" (Day must be two digits)
I believe the problem is just your string format, which should be
"2010-04-04T03:16:04.000000". See the documentation for dateTime for more information.
I don't know anything about the Times ten database, or whether you retrieve values in a "rich" format which you happen to be formatting to a string (in which case you should be able to specify a different format) but I believe that's what's wrong.
I'm trying to write a small javascript code to query data filtered by date.
If I write the following sentence in my browser, I can get data :
"http://www.google.com/fusiontables/gvizdata?tq=SELECT Date, Poids FROM 3049883"
but if I write the same thing, except I want only data after a certain date :
"http://www.google.com/fusiontables/gvizdata?tq=SELECT Date, Poids FROM 3049883 WHERE Date > 2/29/12"
From the SQL-like API, https://developers.google.com/fusiontables/docs/developers_reference#Select, it should work
I get an error witch is "'internal_error', message:'Could not parse query'"
-Date is a DATETIME format in my fusion table.
-I've tried different format, but I can not get data.
What am I doing wrong?
Thank you very much for your help.
The Date value must be quoted and the format is MM/dd/yy so you must pad single digits with leading zeros.
I had success with:
select Date,Poids from 3049883 where Date >= '02/29/12'
Note: I did not test with gvizdata, just with the FT JSONP API