So Im pretty new to this stuff but working through a few issues. What I am trying to do is pull source files from a Flat File Source but the dates in all my source files are formatted to YYYYMMDD so I have inserted a Derived Column task and created an expression to format all the columns with dates YYYYMMDD to YYYY-MM-DD and that looks like this:
LEFT(ISSUE_DT,4) + "-" + SUBSTRING(ISSUE_DT,5,2) + "-" + RIGHT(ISSUE_DT,2)
All is good with that except it's in the data type of DT_WSTR so I dropped in a Columns Conversion task to convert DT_WSTR to DT_DATE but I keep getting the following error:
[Columns Conversion [1]] Error: Data conversion failed while converting
column "ISSUE_DT Formatted" (258) to column "Copy of ISSUE_DT Formatted"
(16). The conversion returned status value 2 and status text "The value
could not be converted because of a potential loss of data.".
I have tried opening the advanced editor and navigated to the Data Conversion Output Columns and tried changing the DataType under Data Type Properties to DT_DATE but still the same error..
What am I missing or doing wrong??
Data Flow
Formatted Dates
Column Conversion
Column Conversion Advanced Editor
You have some dates that are not in the format that your SSIS package is expecting. Perhaps single-digit months or days do not have a leading 0. That scenario would definitely cause this specific error.
Take today for example, if single-digit months or days did not have leading zeroes, you have 2018918 instead of 20180918. Without seeing the data, I can't guarantee that this is the exact issue, but it is something like this. The error is occurring when converting the string to the date. So continuing with the example above, after your Derived Column ISSUED_DT formatted would have a value of '2018-91-18' which of course is not a valid date and causes the error.
Related
I am getting data from rest API in JSON forma and have a scenario where a column can have multiple date format. The current date format could be either 2011-02-12T01:00:00 or 2020-04-15T20:44:57.38or could be null or something else also.
I want to parse it through expression and trying to capture the full date string. The following expression seems to be working fine however it is truncating the millisecond part and returning value upto second only.
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'))
But when I tried it with millisecond usinf below expression:
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'))
It is throwing error:
TT_11132 Transformation [Expression3] had an error evaluating output column [JobStartDate_out].
Error message is [<<Expression Error>> [TO_DATE]: invalid string for converting to Date
... t:TO_DATE(u:TO_CHAR(t:<02/12/2011 01:00:00>,u:'MM/DD/YYYY HH24:MI:SS'),u:'MM/DD/YYYY HH24:MI:SS.MS')].
I searched few option using below but getting parsing error.
DECODE (TRUE,
iff(isnull(%date_milli%),
'\N',
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'),
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'),
ERROR('NOT A VALID DATE')))
What could be the possible resolution to handle the multiple date format in Informatica? Here JSON date format is string and I am mapping it to date/time type and using Output Marco Fields to combine multiple similar column together.
Why dont you check both options - with and without milliseconds?
You can use below iif condition. Also i think your expression has some issues.
I assumed date_milli is a character type. If its a date, then you can change below expressions accordingly.
iff(isnull(%date_milli%),null,
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS')
))
)
I'm attempting to periodically push data to a fusion table, and I have everything working except for the ability to insert the time portion of a DATETIME.
It works as expected if I omit the datetime field all together, e.g.:
service.query().sql(sql='INSERT INTO <my table> (Humidity, Temperature, Luminosity) VALUES (40, 70, 100)')
However, I get a synatx error every time I attempt to write the date. If I separate the date with /'s, it errors on those, so I switched to -'s, which seems to appease it. Only now, it chokes on the :'s that separate hour from minute:
service.query().sql(sql='INSERT INTO <my table> (Date, Humidity, Temperature, Luminosity) VALUES (04-12-2014 13:51, 40, 70, 100)')
yields:
HttpError: <HttpError 400 when requesting https://www.googleapis.com/fusiontables/v1/query?alt=json&sql=INSERT+INTO+(my table id)+%28Date%2C+Humidity%2C+Temperature%2C+Luminosity%29+VALUES+%2804-12-2014+13%3A51%2C+40%2C+70%2C+100%29 returned "Invalid query: Parse error near ':' (line 1, position 117).">
There's only one ':', so it's gotta be the time separator. Is there some other character I should be using to separate hours from minutes? Does it need to be escaped somehow? All of examples I can find in Google's documentation don't interact with DATETIME's.
When I tried without a ':' at all (e.g., 04-12-2014 1351), it didn't throw an error, but the date was interpreted in a wildly inaccurate way (09/01/1351).
Thanks!
Have you tried adding quotation marks around datetime?
Trying to retrieve a dateTime Information from Timesten Database and use it in Saxon Xquery .below is the example for that and getting the below error . Do we need to convert timesten dateTime to saxom dataTime if yes how to do that ? pls help me if have idea.
let $DateVar:=fn:data($PERSON/BIRTHDAY)
where as $PERSON/IN_BIRTHDAY is 2010-04-04 03:16:04.000000
if I am trying
let $day-b-DT :=day-from-dateTime($DateVar)
I am getting
Validation error
FORG0001: Invalid dateTime value "2010-04-04 03:16:04.000000" (Day must be two digits)
net.sf.saxon.s9api.SaxonApiUncheckedException: Invalid dateTime value "2010-04-04 03:16:04.000000" (Day must be two digits)
I believe the problem is just your string format, which should be
"2010-04-04T03:16:04.000000". See the documentation for dateTime for more information.
I don't know anything about the Times ten database, or whether you retrieve values in a "rich" format which you happen to be formatting to a string (in which case you should be able to specify a different format) but I believe that's what's wrong.
I'm trying to write a small javascript code to query data filtered by date.
If I write the following sentence in my browser, I can get data :
"http://www.google.com/fusiontables/gvizdata?tq=SELECT Date, Poids FROM 3049883"
but if I write the same thing, except I want only data after a certain date :
"http://www.google.com/fusiontables/gvizdata?tq=SELECT Date, Poids FROM 3049883 WHERE Date > 2/29/12"
From the SQL-like API, https://developers.google.com/fusiontables/docs/developers_reference#Select, it should work
I get an error witch is "'internal_error', message:'Could not parse query'"
-Date is a DATETIME format in my fusion table.
-I've tried different format, but I can not get data.
What am I doing wrong?
Thank you very much for your help.
The Date value must be quoted and the format is MM/dd/yy so you must pad single digits with leading zeros.
I had success with:
select Date,Poids from 3049883 where Date >= '02/29/12'
Note: I did not test with gvizdata, just with the FT JSONP API
I have made a huge mistake.
Initially I created my model with a field called start_date and made it a string to keep track of event dates.
Now I'm realizing it would be nice to have this field as a date type so I could do calculations like find events where start_date is between today and 1 month from now.
This issue is I already have 500 records so starting over would suck....
The format of the start_date field is in a rails compatible type " 2011-02-21 22:00:00 " but its just a string...
Is there anything I can do?
Create a migration to add a start_date_2 column of the type you want
Model.find(:all).each { |i| i.update_attributes(:start_date_2, Date.new(i.start_date)) }
Create a migration to delete start_date and to rename start_date_2 to start_date
This should work, out of the top of my head.
You could try just doing an EXPORT on the table (making sure to only export data, do not include CREATE and/or DROP table commands).
Create a migration to change the datatype
TRUNCATE the table
IMPORT the data
Since the column is now a date field, it should parse the input of a string just fine, considering that's what you provide it anyway
Perhaps, you can do away with the risk of changing column type if there is live data. The parse methods can save you. From Ruby-doc:
parse(str='-4712-01-01', comp=true, sg=ITALY)
Create a new Date object by parsing from a String, without specifying the format.
str is a String holding a date representation. comp specifies whether to interpret 2-digit years as 19XX (>= 69) or 20XX (< 69); the default is not to. The method will attempt to parse a date from the String using various heuristics; see _parse in date/format.rb for more details. If parsing fails, an ArgumentError will be raised.
Here and here are some more examples / explanations. Hope this helps.